2022-03-10T10:35:34.928 INFO:root:teuthology version: 1.1.0-ff5db916
2022-03-10T10:35:34.929 INFO:teuthology.results:seed: '3607'
2022-03-10T11:03:56.743 INFO:root:teuthology version: 1.1.0-ff5db916
2022-03-10T11:03:56.748 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-10T15:59:17.948 INFO:teuthology.results:Tests finished! gathering results...
2022-03-10T15:59:19.585 INFO:scrape:Found 16 jobs
2022-03-10T15:59:19.588 INFO:scrape:Found 1 distinct failure reasons
2022-03-10T15:59:19.588 INFO:scrape:Failure: wait_for_recovery: failed before timeout expired
2022-03-10T15:59:19.589 INFO:scrape:1 jobs: ['6728833']
2022-03-10T15:59:19.589 INFO:scrape:suites: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_18.04}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2022-03-10T15:59:19.590 INFO:scrape:
2022-03-10T15:59:19.591 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  15


Fail
=================================================================
[6728833]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
-----------------------------------------------------------------
time:   03:11:41
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728833/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728833/
sentry: https://sentry.ceph.com/organizations/ceph/?query=86b1a79b5eb441328db69b33a5e8e2a2

    wait_for_recovery: failed before timeout expired



Pass
=================================================================
[6728831] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:05:38
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728831/

[6728835] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
time:   00:50:17
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728835/

[6728837] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   01:56:22
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728837/

[6728838] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:04:47
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728838/

[6728840] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   02:28:29
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728840/

[6728842] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   00:47:15
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728842/

[6728844] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   01:57:50
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728844/

[6728846] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:06:10
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728846/

[6728847] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
time:   04:58:53
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728847/

[6728849] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04}
time:   00:50:17
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728849/

[6728851] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   01:58:07
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728851/

[6728853] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_18.04}
time:   02:13:04
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728853/

[6728854] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   02:46:16
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728854/

[6728856] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_18.04}
time:   00:47:19
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728856/

[6728858] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_18.04}
time:   01:58:54
info:   http://pulpito.ceph.com/teuthology-2022-03-10_02:30:03-upgrade:mimic-x-octopus-distro-default-smithi/6728858/