2021-06-18T20:58:20.717 INFO:root:teuthology version: 1.1.0-f359b10d
2021-06-18T20:58:20.719 INFO:teuthology.results:subset: '21/1000'
2021-06-18T20:58:20.719 INFO:teuthology.results:seed: '3243'
2021-06-18T22:29:43.769 INFO:root:teuthology version: 1.1.0-f359b10d
2021-06-18T22:29:43.803 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-06-19T09:54:28.074 INFO:teuthology.results:Tests finished! gathering results...
2021-06-19T09:54:30.066 INFO:scrape:Found 16 jobs
2021-06-19T09:54:30.085 INFO:scrape:Found 2 distinct failure reasons
2021-06-19T09:54:30.086 INFO:scrape:Failure: failed to recover before timeout expired
2021-06-19T09:54:30.087 INFO:scrape:2 jobs: ['6179603', '6179591']
2021-06-19T09:54:30.087 INFO:scrape:suites intersection: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2021-06-19T09:54:30.088 INFO:scrape:suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/centos_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2021-06-19T09:54:30.088 INFO:scrape:
2021-06-19T09:54:30.088 INFO:scrape:Dead: b'2021-06-19T09:50:41.412 DEBUG:teuthology.exit:Finished running handlers'
2021-06-19T09:54:30.089 INFO:scrape:3 jobs: ['6179600', '6179597', '6179594']
2021-06-19T09:54:30.089 INFO:scrape:suites intersection: ['1-ceph-install/mimic', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2021-06-19T09:54:30.090 INFO:scrape:suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-nautilus', '6.1-msgr2', '7-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw-swift-ragweed_check', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/rhel_7', 'supported-all-distro/ubuntu_16.04', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2021-06-19T09:54:30.091 INFO:scrape:
2021-06-19T09:54:30.093 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/
failed:  2
dead:    3
running: 0
waiting: 0
queued:  0
passed:  11


Fail
=================================================================
[6179591]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health}
-----------------------------------------------------------------
time:   06:19:16
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179591/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179591/
sentry: https://sentry.ceph.com/organizations/ceph/?query=60578a79b6984ff88ae126f02ea089d6

    failed to recover before timeout expired

[6179603]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health}
-----------------------------------------------------------------
time:   04:27:52
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179603/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179603/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f27c3bba7522431d987faa49bc3f7e09

    failed to recover before timeout expired



Dead
=================================================================
[6179594]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179594/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179594/

    hit max job timeout

[6179597]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179597/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179597/

    hit max job timeout

[6179600]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-nautilus 6.1-msgr2 7-final-workload/{rbd-python rgw-swift-ragweed_check snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179600/
log:    http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179600/

    hit max job timeout



Pass
=================================================================
[6179590] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/bluestore-bitmap supported-all-distro/centos_latest}
time:   02:34:35
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179590/

[6179592] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/bluestore-bitmap supported-all-distro/centos_latest thrashosds-health}
time:   00:55:25
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179592/

[6179593] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/filestore-xfs supported-all-distro/rhel_7}
time:   02:32:14
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179593/

[6179595] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/filestore-xfs supported-all-distro/rhel_7 thrashosds-health}
time:   00:48:27
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179595/

[6179596] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04}
time:   02:30:55
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179596/

[6179598] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/bluestore-bitmap supported-all-distro/ubuntu_16.04 thrashosds-health}
time:   00:45:47
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179598/

[6179599] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/filestore-xfs supported-all-distro/ubuntu_latest}
time:   02:23:45
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179599/

[6179601] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
time:   00:42:09
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179601/

[6179602] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/filestore-xfs supported-all-distro/centos_latest}
time:   02:18:40
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179602/

[6179604] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 7-final-workload objectstore/filestore-xfs supported-all-distro/centos_latest thrashosds-health}
time:   00:45:10
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179604/

[6179605] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-msgr2 4-nautilus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check rgw_swift} objectstore/bluestore-bitmap supported-all-distro/rhel_7}
time:   02:34:54
info:   http://pulpito.ceph.com/yuriw-2021-06-18_15:55:30-upgrade:mimic-x-nautilus-distro-basic-smithi/6179605/