2023-09-07T06:23:05.613 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-07T06:23:05.614 INFO:teuthology.results:subset: '425/1024' 2023-09-07T06:23:05.614 INFO:teuthology.results:seed: '985' 2023-09-07T09:31:19.523 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-07T09:31:19.548 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-07T18:42:55.892 INFO:teuthology.results:Tests finished! gathering results... 2023-09-07T18:42:57.791 INFO:scrape:Found 14 jobs 2023-09-07T18:42:59.070 INFO:scrape:Found 2 distinct failure reasons 2023-09-07T18:42:59.070 INFO:scrape:Dead: b'2023-09-07T18:39:15.445 DEBUG:teuthology.exit:Finished running handlers' 2023-09-07T18:42:59.071 INFO:scrape:1 jobs: ['7390587'] 2023-09-07T18:42:59.071 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-07T18:42:59.072 INFO:scrape: 2023-09-07T18:42:59.072 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2023-09-07T18:42:59.072 INFO:scrape:1 jobs: ['7390582'] 2023-09-07T18:42:59.073 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-09-07T18:42:59.073 INFO:scrape: 2023-09-07T18:42:59.074 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/ failed: 1 dead: 1 running: 0 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [7390582] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 03:44:44 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390582/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c3c3e09bfa54920a14931f8aad51caa Command failed (workunit test suites/fsstress.sh) on smithi153 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=92a7e05e5164798454d327c8a595c2aaf0b88910 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' Dead ================================================================= [7390587] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390587/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390587/ hit max job timeout Pass ================================================================= [7390583] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:59 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390583/ [7390584] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:27:20 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390584/ [7390585] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:47 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390585/ [7390586] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:10 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390586/ [7390588] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:05 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390588/ [7390589] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:45 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390589/ [7390590] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:28 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390590/ [7390591] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:01 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390591/ [7390592] fs:upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:26 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390592/ [7390593] fs:upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:18:15 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390593/ [7390594] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:37:37 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390594/ [7390595] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:56 info: http://pulpito.ceph.com/mchangir-2023-09-07_05:59:51-fs:upgrade-wip-mchangir-testing-20230906.152400-testing-default-smithi/7390595/