2020-07-18T03:38:58.726 INFO:root:teuthology version: 1.0.0-10de4c5
2020-07-18T03:38:58.728 INFO:teuthology.results:seed: '7267'
2020-07-18T03:48:03.939 INFO:root:teuthology version: 1.0.0-10de4c5
2020-07-18T03:48:03.955 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-07-18T15:48:24.028 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2020-07-18T15:48:25.579 INFO:scrape:Found 16 jobs
2020-07-18T15:48:25.624 INFO:scrape:Found 2 distinct failure reasons
2020-07-18T15:48:25.625 INFO:scrape:Dead: b'2020-07-18T15:39:26.337 DEBUG:teuthology.exit:Finished running handlers'
2020-07-18T15:48:25.625 INFO:scrape:2 jobs: ['5235992', '5236005']
2020-07-18T15:48:25.626 INFO:scrape:suites intersection: ['1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_latest}']
2020-07-18T15:48:25.626 INFO:scrape:suites union: ['1-ceph-install/mimic', '1-luminous-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2020-07-18T15:48:25.627 INFO:scrape:
2020-07-18T15:48:25.627 INFO:scrape:Failure: "2020-07-18T04:08:04.615569+0000 mon.a (mon.0) 330 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log
2020-07-18T15:48:25.627 INFO:scrape:3 jobs: ['5235996', '5236004', '5236000']
2020-07-18T15:48:25.628 INFO:scrape:suites intersection: ['1-ceph-install/mimic', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-msgr2', '8-final-workload/{rbd-python', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2020-07-18T15:48:25.628 INFO:scrape:suites union: ['1-ceph-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2020-07-18T15:48:25.629 INFO:scrape:
2020-07-18T15:48:25.631 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/
failed:  3
dead:    1
running: 1
waiting: 0
queued:  0
passed:  11


Fail
=================================================================
[5235996]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   03:14:30
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235996/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235996/

    "2020-07-18T04:08:04.615569+0000 mon.a (mon.0) 330 : cluster [ERR] Health
    check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

[5236000]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   05:17:46
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236000/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236000/

    "2020-07-18T06:11:46.507256+0000 mon.c (mon.0) 374 : cluster [ERR] Health
    check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

[5236004]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   03:24:29
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236004/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236004/

    "2020-07-18T04:40:55.958583+0000 mon.b (mon.0) 275 : cluster [ERR] Health
    check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log



Dead
=================================================================
[5235992]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235992/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235992/



Running
=================================================================
[5236005] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest}
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236005/



Pass
=================================================================
[5235991] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:37:24
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235991/

[5235993] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
time:   00:54:01
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235993/

[5235994] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:33:17
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235994/

[5235995] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:40:03
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235995/

[5235997] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest}
time:   00:49:28
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235997/

[5235998] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:34:21
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235998/

[5235999] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:35:59
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5235999/

[5236001] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
time:   00:53:00
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236001/

[5236002] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:32:05
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236002/

[5236003] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:38:09
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236003/

[5236006] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:32:07
info:   http://pulpito.ceph.com/teuthology-2020-07-18_02:30:03-upgrade:mimic-x-octopus-distro-basic-smithi/5236006/