2022-03-09T13:50:06.294 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-09T13:50:06.295 INFO:teuthology.results:subset: '1/32' 2022-03-09T13:50:06.296 INFO:teuthology.results:seed: '4022' 2022-03-09T13:57:31.301 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-09T13:57:31.309 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-09T20:39:01.256 INFO:teuthology.results:Tests finished! gathering results... 2022-03-09T20:39:02.345 INFO:scrape:Found 12 jobs 2022-03-09T20:39:04.828 INFO:scrape:Found 3 distinct failure reasons 2022-03-09T20:39:04.829 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-03-09T20:39:04.829 INFO:scrape:1 jobs: ['6728201'] 2022-03-09T20:39:04.830 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-09T20:39:04.831 INFO:scrape: 2022-03-09T20:39:04.832 INFO:scrape:Dead: b'2022-03-09T20:32:51.671 DEBUG:teuthology.exit:Finished running handlers' 2022-03-09T20:39:04.833 INFO:scrape:3 jobs: ['6728205', '6728199', '6728194'] 2022-03-09T20:39:04.833 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-09T20:39:04.834 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-09T20:39:04.835 INFO:scrape: 2022-03-09T20:39:04.836 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi166 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27bad0cf8decf43ee4c77b0b76e8f0be5095a177 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' 2022-03-09T20:39:04.837 INFO:scrape:1 jobs: ['6728203'] 2022-03-09T20:39:04.838 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-03-09T20:39:04.839 INFO:scrape: 2022-03-09T20:39:04.841 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/ failed: 1 dead: 4 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [6728203] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:22:10 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728203/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ee45dc565474981bb5bb5536740e2ce Command failed (workunit test suites/fsstress.sh) on smithi166 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27bad0cf8decf43ee4c77b0b76e8f0be5095a177 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' Dead ================================================================= [6728194] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728194/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728194/ hit max job timeout [6728199] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728199/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728199/ hit max job timeout [6728201] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:35:25 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728201/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728201/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fcbe92e490b4a79b11444aac389022c Command failed (workunit test suites/fsstress.sh) on smithi184 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27bad0cf8decf43ee4c77b0b76e8f0be5095a177 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6728205] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728205/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728205/ hit max job timeout Pass ================================================================= [6728195] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:53 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728195/ [6728196] fs:upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:14:38 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728196/ [6728197] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:08 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728197/ [6728198] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:14 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728198/ [6728200] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:28:27 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728200/ [6728202] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:31:46 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728202/ [6728204] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:50 info: http://pulpito.ceph.com/vshankar-2022-03-09_13:49:18-fs:upgrade-master-testing-default-smithi/6728204/