2022-02-04T21:45:45.481 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-04T21:45:45.482 INFO:teuthology.results:seed: '7525' 2022-02-04T22:30:06.689 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-04T22:30:06.731 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-04T23:00:08.910 INFO:teuthology.results:Tests finished! gathering results... 2022-02-04T23:00:11.044 INFO:scrape:Found 23 jobs 2022-02-04T23:00:11.147 INFO:scrape:Found 1 distinct failure reasons 2022-02-04T23:00:11.148 INFO:scrape:Failure: Command failed on smithi183 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55fd5d5e-8609-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_4' 2022-02-04T23:00:11.148 INFO:scrape:1 jobs: ['6661788'] 2022-02-04T23:00:11.149 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-02-04T23:00:11.149 INFO:scrape: 2022-02-04T23:00:11.151 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 22 Fail ================================================================= [6661788] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661788/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661788/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c47a592d550a434389edc096ee2f0f45 Command failed on smithi183 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 55fd5d5e-8609-11ec-8c35-001a4aab830c -- ceph orch daemon add osd smithi183:vg_nvme/lv_4' Pass ================================================================= [6661775] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:22:16 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661775/ [6661776] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:30 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661776/ [6661777] fs:upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:07:37 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661777/ [6661778] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:46 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661778/ [6661779] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:24:59 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661779/ [6661780] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:15:43 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661780/ [6661781] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:13 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661781/ [6661782] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:57 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661782/ [6661783] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:26:37 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661783/ [6661784] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:18:39 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661784/ [6661785] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:20 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661785/ [6661786] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:21 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661786/ [6661787] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:23:04 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661787/ [6661789] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:41 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661789/ [6661790] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:24 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661790/ [6661791] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:15:45 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661791/ [6661792] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:10 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661792/ [6661793] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:27:47 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661793/ [6661794] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:25:55 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661794/ [6661795] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:20:44 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661795/ [6661796] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:28:26 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661796/ [6661797] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:24:30 info: http://pulpito.ceph.com/pdonnell-2022-02-04_21:45:38-fs:upgrade-wip-pdonnell-testing-20220204.020350-distro-default-smithi/6661797/