2022-08-27T13:22:15.819 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-08-27T13:22:15.820 INFO:teuthology.results:seed: '1467'
2022-08-27T13:36:41.440 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-08-27T13:36:41.442 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-08-27T13:51:42.830 INFO:teuthology.results:Tests finished! gathering results...
2022-08-27T13:51:44.416 INFO:scrape:Found 16 jobs
2022-08-27T13:51:45.141 INFO:scrape:Found 1 distinct failure reasons
2022-08-27T13:51:45.143 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f2877ae32a72fc25acadef57597f44988b805c38
2022-08-27T13:51:45.144 INFO:scrape:16 jobs: ['6991221', '6991216', '6991222', '6991230', '6991218', '6991223', '6991228', '6991227', '6991225', '6991229', '6991215', '6991220', '6991224', '6991219', '6991226', '6991217']
2022-08-27T13:51:45.145 INFO:scrape:suites intersection: ['start}', 'ubuntu_18.04}']
2022-08-27T13:51:45.147 INFO:scrape:suites union: ['1-ceph-install/mimic', '1-luminous-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '4-ec-workload', '4-octopus', '4-workload/{radosbench', '5-finish-upgrade', '5-msgr2', '6-final-workload/{blogbench', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:mimic-x/parallel/{0-cluster/{openstack', 'upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2022-08-27T13:51:45.147 INFO:scrape:
2022-08-27T13:51:45.151 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/
failed:  16
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6991215]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:12:20
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991215/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991215/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2e463aec5af4e27aa7bca8eb6c885ae

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991216]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:14:07
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991216/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991216/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7b5c0e6fc91f480b820954d8dde9a13e

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991217]  upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:14:46
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991217/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991217/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d53d3632e16644c3885f44d4548f1333

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991218]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:12:54
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991218/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991218/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a3ae47c6fdd4a9f80830192531ef7a3

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991219]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:12:05
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991219/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991219/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eed5a48ad1174b07bc1eea5281c7eb55

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991220]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:14:16
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991220/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991220/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9787b19b164d4d549ed408cc1e52f390

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991221]  upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:12:51
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991221/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991221/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8ecfeb3018ac4f91b1ab4b1519eda17b

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991222]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:11:16
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991222/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991222/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2a425eeea75c4e01a0721598aaa4a667

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991223]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:10:17
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991223/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991223/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d8554d9159144013a5a8f5f1caa3c747

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991224]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:11:11
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991224/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991224/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b5e6059db5e43df95ef36330e178a44

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991225]  upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:11:19
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991225/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991225/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9abd7d9f650945efb9d3f1691dd632d1

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991226]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:09:30
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991226/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991226/
sentry: https://sentry.ceph.com/organizations/ceph/?query=259e1441fac647fab69b4bbf08b95eb7

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991227]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
-----------------------------------------------------------------
time:   00:08:39
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991227/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991227/
sentry: https://sentry.ceph.com/organizations/ceph/?query=05ded3d75b144668ad4ee1d504ed5134

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991228]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:08:58
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991228/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991228/
sentry: https://sentry.ceph.com/organizations/ceph/?query=795849e4e8284898b10b0a36e5eb081d

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991229]  upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   00:09:01
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991229/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991229/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4452cda1e85c488fa9dfa5db4ae9b64d

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38

[6991230]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
-----------------------------------------------------------------
time:   00:07:54
info:   http://pulpito.ceph.com/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991230/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-08-25_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6991230/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4731b2a6ca724ed495011ad64db735ca

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =f2877ae32a72fc25acadef57597f44988b805c38