2022-03-18T02:58:00.597 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-18T02:58:00.665 INFO:teuthology.results:subset: '12/32' 2022-03-18T02:58:00.666 INFO:teuthology.results:seed: '3865' 2022-03-18T03:04:36.386 INFO:root:teuthology version: 1.1.0-aa078dbe 2022-03-18T03:04:36.390 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-18T09:41:05.695 INFO:teuthology.results:Tests finished! gathering results... 2022-03-18T09:41:06.632 INFO:scrape:Found 12 jobs 2022-03-18T09:41:09.057 INFO:scrape:Found 4 distinct failure reasons 2022-03-18T09:41:09.058 INFO:scrape:Dead: b'2022-03-18T09:37:20.513 DEBUG:teuthology.exit:Finished running handlers' 2022-03-18T09:41:09.059 INFO:scrape:2 jobs: ['6743907', '6743897'] 2022-03-18T09:41:09.059 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.060 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.060 INFO:scrape: 2022-03-18T09:41:09.061 INFO:scrape:Crash: "2022-03-18T03:37:48.927621+0000 mon.a (mon.0) 247 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log 2022-03-18T09:41:09.061 INFO:scrape:ceph version 17.0.0-11165-gc7bc6ea9 (c7bc6ea9ed836057fd41cf9929400b2956af6d42) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7fe8a69bbc20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fe8a79db262] 5: /usr/lib64/ceph/libceph-common.so.2(+0x284425) [0x7fe8a79db425] 6: (SimpleLock::put_xlock()+0x10e) [0x55fa7fde0a1e] 7: (Locker::xlock_finish(std::_Rb_tree_const_iterator const&, MutationImpl*, bool*)+0x94) [0x55fa7fdce2e4] 8: (Locker::_drop_locks(MutationImpl*, std::set, std::allocator >*, bool)+0x20a) [0x55fa7fdce99a] 9: (Locker::drop_non_rdlocks(MutationImpl*, std::set, std::allocator >*)+0x5e) [0x55fa7fdcef8e] 10: (Server::reply_client_request(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x54c) [0x55fa7fc2d42c] 11: (Server::respond_to_request(boost::intrusive_ptr&, int)+0x463) [0x55fa7fc2e243] 12: (Server::_rename_finish(boost::intrusive_ptr&, CDentry*, CDentry*, CDentry*)+0x197) [0x55fa7fc554a7] 13: (MDSContext::complete(int)+0x203) [0x55fa7fefbfc3] 14: (MDSIOContextBase::complete(int)+0x69c) [0x55fa7fefc75c] 15: (MDSLogContextBase::complete(int)+0x44) [0x55fa7fefc9d4] 16: (Finisher::finisher_thread_entry()+0x19d) [0x7fe8a7a8684d] 17: /lib64/libpthread.so.0(+0x817f) [0x7fe8a69b117f] 18: clone() 2022-03-18T09:41:09.062 INFO:scrape:2 jobs: ['6743898', '6743903'] 2022-03-18T09:41:09.062 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.063 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.063 INFO:scrape: 2022-03-18T09:41:09.063 INFO:scrape:Failure: Command failed on smithi196 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 59f53f30-a66b-11ec-8c35-001a4aab830c -- ceph mon dump -f json' 2022-03-18T09:41:09.064 INFO:scrape:1 jobs: ['6743904'] 2022-03-18T09:41:09.064 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.065 INFO:scrape: 2022-03-18T09:41:09.065 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-03-18T09:41:09.066 INFO:scrape:1 jobs: ['6743901'] 2022-03-18T09:41:09.066 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-18T09:41:09.067 INFO:scrape: 2022-03-18T09:41:09.068 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/ failed: 3 dead: 3 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6743898] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:32:51 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743898/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743898/ "2022-03-18T03:37:48.927621+0000 mon.a (mon.0) 247 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log [6743903] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:32:53 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743903/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743903/ "2022-03-18T03:37:42.720635+0000 mon.a (mon.0) 230 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.a" in cluster log [6743904] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:17:18 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743904/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc144d675f76454a879b70417ecc9ba7 Command failed on smithi196 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 59f53f30-a66b-11ec-8c35-001a4aab830c -- ceph mon dump -f json' Dead ================================================================= [6743897] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743897/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743897/ hit max job timeout [6743901] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:36:29 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743901/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2616a2cf14f5405d90d91d3bf6758493 Command failed (workunit test suites/fsstress.sh) on smithi066 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7bc6ea9ed836057fd41cf9929400b2956af6d42 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6743907] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743907/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743907/ hit max job timeout Pass ================================================================= [6743899] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:35:13 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743899/ [6743900] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:45 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743900/ [6743902] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:27:28 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743902/ [6743905] fs:upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:12:38 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743905/ [6743906] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:12 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743906/ [6743908] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:28:44 info: http://pulpito.ceph.com/vshankar-2022-03-18_02:56:29-fs:upgrade-wip-vshankar-testing-20220317-101203-testing-default-smithi/6743908/