2021-06-27T22:44:38.380 INFO:root:teuthology version: 1.1.0-26631ede
2021-06-27T22:44:38.381 INFO:teuthology.results:seed: '3462'
2021-06-28T13:52:18.709 INFO:root:teuthology version: 1.1.0-26631ede
2021-06-28T13:52:18.744 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-06-28T14:12:20.266 INFO:teuthology.results:Tests finished! gathering results...
2021-06-28T14:12:23.123 INFO:scrape:Found 16 jobs
2021-06-28T14:12:27.513 INFO:scrape:Found 1 distinct failure reasons
2021-06-28T14:12:27.525 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=74ba08b1cab1ab1a3b9f2fbe200c70898e373889
2021-06-28T14:12:27.525 INFO:scrape:16 jobs: ['6166430', '6166439', '6166437', '6166441', '6166434', '6166435', '6166440', '6166432', '6166427', '6166433', '6166442', '6166438', '6166436', '6166431', '6166429', '6166428']
2021-06-28T14:12:27.526 INFO:scrape:suites intersection: ['start}', 'ubuntu_latest}']
2021-06-28T14:12:27.527 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '2-workload/{blogbench', '3-thrash/default', '3-upgrade-sequence/upgrade-all', '3-upgrade-sequence/upgrade-mon-osd-mds', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-octopus', '4-workload/{radosbench', '5-final-workload/{blogbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'ec-rados-default', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rados-snaps-few-objects', 'rados_api', 'rados_loadgenbig', 'rados_loadgenmix', 'rados_mon_thrash', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'rbd_cls', 'rbd_import_export', 'readwrite', 'rgw', 'rgw_ragweed_check}', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'test_rbd_api', 'test_rbd_python}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/parallel/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack']
2021-06-28T14:12:27.527 INFO:scrape:
2021-06-28T14:12:27.529 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba
=================================================================
info:    http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/
failed:  16
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6166427]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:57
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166427/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166427/
sentry: https://sentry.ceph.com/organizations/ceph/?query=766719cfe3b1439d9109a2676e7ca60f

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166428]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:09:05
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166428/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166428/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6fb326fdda3d4309b53d47482c3177b0

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166429]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:44
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166429/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166429/
sentry: https://sentry.ceph.com/organizations/ceph/?query=20880b6c3e1f4494bf5deb00bc130f19

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166430]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:12
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166430/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166430/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7e100d51e92b454fb52d33a649d1c889

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166431]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:29
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166431/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166431/
sentry: https://sentry.ceph.com/organizations/ceph/?query=17b2433789294444bcff16860e3bca40

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166432]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:13:32
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166432/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166432/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4dd5cfa0596d4b7cbe30449b02bce924

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166433]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:54
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166433/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166433/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c20e38a6a5854d1c8e4cbb8fdbbde9b7

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166434]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:05
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166434/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166434/
sentry: https://sentry.ceph.com/organizations/ceph/?query=826d3dbd10aa44dcae3e8ee6cfa5c63a

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166435]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:10:21
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166435/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166435/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f01849b8dd314d1eb5525e1beff2a4a7

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166436]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:15
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166436/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166436/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e0f95472c8d14b28bfaede81d63c6f68

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166437]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:15:50
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166437/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166437/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eab30d0616754b06a350ad1883be682c

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166438]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:12:10
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166438/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166438/
sentry: https://sentry.ceph.com/organizations/ceph/?query=04790272aa924a68986bf9753e51d65a

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166439]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:07:33
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166439/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166439/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fac129102a38423fb72737166ff1ba16

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166440]  upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:08:17
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166440/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166440/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1c8b1870d90247218ca2a54a8a4799f6

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166441]  upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:09:22
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166441/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166441/
sentry: https://sentry.ceph.com/organizations/ceph/?query=81cf9bad13da4456931ff009545d7f9e

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889

[6166442]  upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
-----------------------------------------------------------------
time:   00:06:39
info:   http://pulpito.ceph.com/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166442/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2021-06-11_14:23:01-upgrade:nautilus-x-octopus-distro-basic-gibba/6166442/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c512e55009e34c7ea476290a38d8c622

    Failed to fetch package version from https://shaman.ceph.com/api/search/?st
    atus=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1
    =74ba08b1cab1ab1a3b9f2fbe200c70898e373889