2020-07-14T02:30:58.048 INFO:root:teuthology version: 1.0.0-da68153
2020-07-14T02:30:58.050 INFO:teuthology.results:seed: '587'
2020-07-14T02:44:44.343 INFO:root:teuthology version: 1.0.0-da68153
2020-07-14T02:44:44.368 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-07-14T08:07:51.039 INFO:teuthology.results:Tests finished! gathering results...
2020-07-14T08:07:52.732 INFO:scrape:Found 16 jobs
2020-07-14T08:07:52.797 INFO:scrape:Found 2 distinct failure reasons
2020-07-14T08:07:52.799 INFO:scrape:Failure: "2020-07-14T03:09:52.090822+0000 mon.a (mon.0) 437 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log
2020-07-14T08:07:52.799 INFO:scrape:2 jobs: ['5224903', '5224909']
2020-07-14T08:07:52.800 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_latest}']
2020-07-14T08:07:52.800 INFO:scrape:suites union: ['1-ceph-install/mimic', '1-luminous-install/mimic', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:mimic-x/stress-split/{0-cluster/{openstack']
2020-07-14T08:07:52.801 INFO:scrape:
2020-07-14T08:07:52.802 INFO:scrape:Failure: reached maximum tries (60) after waiting for 900 seconds
2020-07-14T08:07:52.802 INFO:scrape:1 jobs: ['5224907']
2020-07-14T08:07:52.802 INFO:scrape:
2020-07-14T08:07:52.805 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/
failed:  2
dead:    1
running: 0
waiting: 0
queued:  0
passed:  13


Fail
=================================================================
[5224903]  upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   00:53:22
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224903/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224903/

    "2020-07-14T03:09:52.090822+0000 mon.a (mon.0) 437 : cluster [ERR] Health
    check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

[5224909]  upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
-----------------------------------------------------------------
time:   03:14:23
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224909/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224909/

    "2020-07-14T03:14:50.996923+0000 mon.a (mon.0) 355 : cluster [ERR] Health
    check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log



Dead
=================================================================
[5224907]  upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224907/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224907/
sentry: http://sentry.ceph.com/sepia/teuthology/?q=6857c27449e94a178cd3ae0a0c40affb

    reached maximum tries (60) after waiting for 900 seconds



Pass
=================================================================
[5224885] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:36:20
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224885/

[5224886] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
time:   05:09:28
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224886/

[5224888] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
time:   00:51:47
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224888/

[5224889] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:30:23
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224889/

[5224891] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:35:47
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224891/

[5224893] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest}
time:   03:24:55
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224893/

[5224895] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest}
time:   00:50:04
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224895/

[5224897] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:32:24
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224897/

[5224899] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest}
time:   02:34:52
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224899/

[5224901] upgrade:mimic-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest}
time:   04:53:28
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224901/

[5224905] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:29:40
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224905/

[5224911] upgrade:mimic-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-luminous-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-ec-workload 5-finish-upgrade 6-octopus 7-final-workload objectstore/filestore-xfs thrashosds-health ubuntu_latest}
time:   00:52:28
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224911/

[5224914] upgrade:mimic-x/parallel/{0-cluster/{openstack start} 1-ceph-install/mimic 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-msgr2 6-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest}
time:   02:30:44
info:   http://pulpito.ceph.com/teuthology-2020-07-14_02:30:02-upgrade:mimic-x-octopus-distro-basic-smithi/5224914/