2023-12-11T12:51:25.674 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-11T12:51:25.674 INFO:teuthology.results:subset: '48/1024' 2023-12-11T12:51:25.674 INFO:teuthology.results:seed: '2541' 2023-12-11T13:01:59.645 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-11T13:01:59.683 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-11T16:57:17.351 INFO:teuthology.results:Tests finished! gathering results... 2023-12-11T16:57:18.857 INFO:scrape:Found 16 jobs 2023-12-11T16:57:21.748 INFO:scrape:Found 2 distinct failure reasons 2023-12-11T16:57:21.749 INFO:scrape:Failure: Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b99a83e-9828-11ee-95a3-87774f69a715 -- ceph mon dump -f json' 2023-12-11T16:57:21.749 INFO:scrape:4 jobs: ['7486891', '7486899', '7486888', '7486896'] 2023-12-11T16:57:21.749 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-11T16:57:21.749 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-11T16:57:21.749 INFO:scrape: 2023-12-11T16:57:21.750 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2023-12-11T16:57:21.750 INFO:scrape:1 jobs: ['7486894'] 2023-12-11T16:57:21.751 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/quincy'] 2023-12-11T16:57:21.751 INFO:scrape: 2023-12-11T16:57:21.752 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [7486888] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:37:44 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486888/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b6c133d2be345378c5acc947b1cb3c2 Command failed on smithi026 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cdc157dc-9827-11ee-95a3-87774f69a715 -e sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7486891] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:18:23 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486891/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486891/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff517c8d0e374c2f8679f3f468cd5068 Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b99a83e-9828-11ee-95a3-87774f69a715 -- ceph mon dump -f json' [7486894] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:45:34 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486894/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ef16f6cec79144e3867f556588ae71f8 Command failed (workunit test suites/fsstress.sh) on smithi195 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [7486896] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:54 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486896/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486896/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e152b8f5d97740f48b6c441d95a32a0b Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 765382ee-9828-11ee-95a3-87774f69a715 -e sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [7486899] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:36:00 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486899/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91b2db7f975b482ebd177b166805dc1a Command failed on smithi115 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d81c5eb0-9828-11ee-95a3-87774f69a715 -e sha1=f1fbf3c583d03e19f5e58eb618e78d8e10d075f9 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' Pass ================================================================= [7486887] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:39 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486887/ [7486889] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:23:54 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486889/ [7486890] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:32 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486890/ [7486892] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:40 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486892/ [7486893] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:54:21 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486893/ [7486895] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:21 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486895/ [7486897] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:23 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486897/ [7486898] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:57:11 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486898/ [7486900] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:45 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486900/ [7486901] fs:upgrade/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:18 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486901/ [7486902] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:14:53 info: https://pulpito.ceph.com/mchangir-2023-12-11_12:51:15-fs:upgrade-wip-mchangir-testing-20231205.064351-testing-default-smithi/7486902/