2022-01-31T19:13:57.631 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-31T19:13:57.633 INFO:teuthology.results:seed: '8610' 2022-01-31T19:25:11.836 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-31T19:25:11.842 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-31T23:13:28.717 INFO:teuthology.results:Tests finished! gathering results... 2022-01-31T23:13:31.308 INFO:scrape:Found 23 jobs 2022-01-31T23:13:32.539 INFO:scrape:Found 3 distinct failure reasons 2022-01-31T23:13:32.539 INFO:scrape:Dead: b'2022-01-31T23:11:10.943 DEBUG:teuthology.exit:Finished running handlers' 2022-01-31T23:13:32.540 INFO:scrape:5 jobs: ['6651572', '6651583', '6651577', '6651580', '6651574'] 2022-01-31T23:13:32.540 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-inline/yes', '3-upgrade-with-workload', '3-verify', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.541 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/yes', '3-upgrade-with-workload', '3-verify', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.541 INFO:scrape: 2022-01-31T23:13:32.542 INFO:scrape:Failure: Command failed on smithi074 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4799024-82cd-11ec-8c35-001a4aab830c -- bash -c \'ceph --format=json mds versions | jq -e ". | add == 4"\'' 2022-01-31T23:13:32.543 INFO:scrape:4 jobs: ['6651588', '6651585', '6651567', '6651569'] 2022-01-31T23:13:32.543 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '3-verify', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.544 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '3-verify', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.544 INFO:scrape: 2022-01-31T23:13:32.545 INFO:scrape:Failure: Command failed on smithi114 with status 1: 'sudo yum install -y kernel' 2022-01-31T23:13:32.545 INFO:scrape:7 jobs: ['6651586', '6651575', '6651568', '6651566', '6651578', '6651582', '6651571'] 2022-01-31T23:13:32.546 INFO:scrape:suites intersection: ['centos_latest', 'conf/{client', 'mds', 'mon', 'osd}', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.547 INFO:scrape:suites union: ['1-client', '1-upgrade}}', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs:upgrade/nofs/{bluestore-bitmap', 'mds', 'mon', 'no-mds-cluster', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'overrides/{pg-warn', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-01-31T23:13:32.547 INFO:scrape: 2022-01-31T23:13:32.549 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/ failed: 11 dead: 5 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [6651566] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651566/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1f320f21424c4375990c6587894a4a80 Command failed on smithi124 with status 1: 'sudo yum install -y kernel' [6651567] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:17:44 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651567/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33b4fd157b4e4da49a9478b9edc4842a Command failed on smithi061 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 478b3fba-82cc-11ec-8c35-001a4aab830c -- bash -c 'ceph fs dump'" [6651568] fs:upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651568/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651568/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0ff7a2f752e439ca7ae65e6a76becbd Command failed on smithi107 with status 1: 'sudo yum install -y kernel' [6651569] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651569/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb130cd3a0f7424294abbf51a577bd60 Command failed on smithi190 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4ce55806-82cc-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi190:vg_nvme/lv_4' [6651571] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651571/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651571/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77cecc6c51314a72954e6af54e37d910 Command failed on smithi060 with status 1: 'sudo yum install -y kernel' [6651575] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651575/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dac0c52a35954c34a7e09ceef6e04701 Command failed on smithi068 with status 1: 'sudo yum install -y kernel' [6651578] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651578/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651578/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6604783d59944c4ca024df6e7cf5153e Command failed on smithi157 with status 1: 'sudo yum install -y kernel' [6651582] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651582/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd1bcbb5256e4a18be9415d064ea4207 Command failed on smithi124 with status 1: 'sudo yum install -y kernel' [6651585] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:18:14 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651585/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651585/ sentry: https://sentry.ceph.com/organizations/ceph/?query=841ffba641cc40248743971ab62b426f Command failed on smithi055 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5ec9f698-82cd-11ec-8c35-001a4aab830c -- bash -c \'ceph --format=json mds versions | jq -e ". | add == 4"\'' [6651586] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651586/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=973c87102b434776a29ff60265b7565d Command failed on smithi114 with status 1: 'sudo yum install -y kernel' [6651588] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:18:10 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651588/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651588/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19775ec92fab4aefa641b39088ac40b8 Command failed on smithi074 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d4799024-82cd-11ec-8c35-001a4aab830c -- bash -c \'ceph --format=json mds versions | jq -e ". | add == 4"\'' Dead ================================================================= [6651572] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651572/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651572/ [6651574] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651574/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651574/ [6651577] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651577/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651577/ [6651580] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651580/ [6651583] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651583/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651583/ Pass ================================================================= [6651570] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:49 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651570/ [6651573] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:08 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651573/ [6651576] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:09 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651576/ [6651579] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:44 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651579/ [6651581] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:55 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651581/ [6651584] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:29 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651584/ [6651587] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 3-verify 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:36 info: http://pulpito.ceph.com/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651587/