2022-04-27T09:01:25.425 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T09:01:25.426 INFO:teuthology.results:subset: '27/32' 2022-04-27T09:01:25.426 INFO:teuthology.results:seed: '2092' 2022-04-27T09:12:02.359 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T09:12:02.362 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-27T15:51:32.105 INFO:teuthology.results:Tests finished! gathering results... 2022-04-27T15:51:33.313 INFO:scrape:Found 12 jobs 2022-04-27T15:51:35.885 INFO:scrape:Found 4 distinct failure reasons 2022-04-27T15:51:35.926 INFO:scrape:Dead: b'2022-04-27T15:42:56.337 DEBUG:teuthology.exit:Finished running handlers' 2022-04-27T15:51:35.926 INFO:scrape:ceph version 17.0.0-11808-g5ba7d852 (5ba7d8520d02dbf876b7c07fd978e9987e1795c8) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f46f3c12ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f46f5485182] 5: /usr/lib64/ceph/libceph-common.so.2(+0x285345) [0x7f46f5485345] 6: (CDir::check_rstats(bool)+0x17dd) [0x55ad1103a5ed] 7: (MDCache::predirty_journal_parents(boost::intrusive_ptr, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x12a9) [0x55ad10ede799] 8: (MDCache::_create_system_file(CDir*, std::basic_string_view >, CInode*, MDSContext*)+0x64e) [0x55ad10ee012e] 9: (MDCache::populate_mydir()+0x846) [0x55ad10f232d6] 10: (MDCache::open_root()+0xd8) [0x55ad10f23618] 11: (C_MDS_RetryOpenRoot::finish(int)+0x27) [0x55ad10f92e07] 12: (MDSContext::complete(int)+0x203) [0x55ad110fbc13] 13: (MDSRank::_advance_queues()+0x84) [0x55ad10dda764] 14: (MDSRank::ProgressThread::entry()+0xc5) [0x55ad10ddae95] 15: /lib64/libpthread.so.0(+0x81cf) [0x7f46f3c081cf] 16: clone() 2022-04-27T15:51:35.926 INFO:scrape:2 jobs: ['6808582', '6808588'] 2022-04-27T15:51:35.926 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.927 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.927 INFO:scrape: 2022-04-27T15:51:35.927 INFO:scrape:Dead: b'2022-04-27T15:42:54.713 DEBUG:teuthology.exit:Finished running handlers' 2022-04-27T15:51:35.927 INFO:scrape:1 jobs: ['6808583'] 2022-04-27T15:51:35.927 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.927 INFO:scrape: 2022-04-27T15:51:35.927 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-04-27T15:51:35.927 INFO:scrape:1 jobs: ['6808580'] 2022-04-27T15:51:35.928 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.928 INFO:scrape: 2022-04-27T15:51:35.928 INFO:scrape:Failure: Extra data: line 2 column 554 (char 554) 2022-04-27T15:51:35.928 INFO:scrape:2 jobs: ['6808584', '6808578'] 2022-04-27T15:51:35.928 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.928 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T15:51:35.928 INFO:scrape: 2022-04-27T15:51:35.929 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/ failed: 3 dead: 3 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6808578] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:25:48 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808578/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808578/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1c0772977dd4e048f771adcccb6358c Extra data: line 2 column 557 (char 557) [6808580] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:38:18 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808580/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3188d11caa74fb2abc9395cdc901c66 Command failed (workunit test suites/fsstress.sh) on smithi150 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5faec6ad8cc6050dab5d3cff07b1f32ae8f6c4b5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6808584] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:25:30 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808584/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808584/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6aaacdf929f84e46ae779c117875c97d Extra data: line 2 column 554 (char 554) Dead ================================================================= [6808582] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808582/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808582/ hit max job timeout [6808583] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808583/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808583/ hit max job timeout [6808588] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808588/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808588/ hit max job timeout Pass ================================================================= [6808579] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:24 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808579/ [6808581] fs:upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:15:01 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808581/ [6808585] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:40:04 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808585/ [6808586] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:26 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808586/ [6808587] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:42:10 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808587/ [6808589] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:39:50 info: http://pulpito.ceph.com/vshankar-2022-04-27_09:00:32-fs:upgrade-master-testing-default-smithi/6808589/