2021-03-28T09:50:18.451 INFO:root:teuthology version: 1.1.0-d5a60f86
2021-03-28T09:50:18.456 INFO:teuthology.results:seed: '7369'
2021-03-28T10:49:29.675 INFO:root:teuthology version: 1.1.0-d5a60f86
2021-03-28T10:49:29.715 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-03-28T11:06:22.160 INFO:teuthology.results:Tests finished! gathering results...
2021-03-28T11:06:23.454 INFO:scrape:Found 16 jobs
2021-03-28T11:06:25.583 INFO:scrape:Found 1 distinct failure reasons
2021-03-28T11:06:25.584 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5e61b15880d42cb4248c17ea9494bc11be2b7a78
2021-03-28T11:06:25.584 INFO:scrape:16 jobs: ['5958722', '5958721', '5958717', '5958718', '5958713', '5958724', '5958712', '5958715', '5958719', '5958720', '5958714', '5958723', '5958727', '5958725', '5958726', '5958716']
2021-03-28T11:06:25.585 INFO:scrape:suites intersection: ['start}', 'ubuntu_latest}']
2021-03-28T11:06:25.585 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-octopus', '4-workload/{radosbench', '5-final-workload/{blogbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack']
2021-03-28T11:06:25.586 INFO:scrape:
2021-03-28T11:06:25.588 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba
=================================================================
info:    http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/
failed:  16
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[5958712]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:35
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958712/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958712/
sentry: https://sentry.ceph.com/organizations/ceph/?query=78eb5b98c8a441a39db2b58b47e745bb

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958713]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:09:08
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958713/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958713/
sentry: https://sentry.ceph.com/organizations/ceph/?query=15568b25ef7642f898ebd350817d9f17

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958714]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:09:19
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958714/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958714/
sentry: https://sentry.ceph.com/organizations/ceph/?query=24e006b0f03845389144d010697985d0

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958715]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:05:59
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958715/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958715/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44e70375087c40b6be1695ac98cf95ea

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958716]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:43
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958716/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958716/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c268d305f7d42a7bd3610f1ce086363

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958717]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:46
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958717/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958717/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c4d338ad69447a5ba31b9c5da0ac021

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958718]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:22
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958718/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958718/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b04d9ede961b41bbaf7f3f5a3ed82da7

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958719]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:05:56
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958719/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958719/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a1897247385439ebdf7955a99cccc0c

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958720]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:54
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958720/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958720/
sentry: https://sentry.ceph.com/organizations/ceph/?query=25f877b0e2024b1fb11889b17d34e3c8

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958721]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:43
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958721/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958721/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9af300dcc8f74117be3669931a116da6

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958722]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:36
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958722/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958722/
sentry: https://sentry.ceph.com/organizations/ceph/?query=457bf33d32954907b937629fa7ebd764

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958723]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:05:47
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958723/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958723/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ea6c09ecd0e54cf5ae89137f0a543955

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958724]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:37
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958724/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958724/
sentry: https://sentry.ceph.com/organizations/ceph/?query=389fea31b10d4449a82ac68477a949aa

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958725]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:45
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958725/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958725/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2b261d8fa589490a92c59233c758adf1

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958726]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:16
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958726/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958726/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da363eb3bbf54ff084bc78e93d7628c8

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78

[5958727]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:22
info:   http://pulpito.ceph.com/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958727/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-12_14:23:03-upgrade:nautilus-x-octopus-distro-basic-gibba/5958727/
sentry: https://sentry.ceph.com/organizations/ceph/?query=864a1bab114d4a5187beeb0d1fe91f56

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =5e61b15880d42cb4248c17ea9494bc11be2b7a78