2020-07-17T14:24:25.015 INFO:root:teuthology version: 1.0.0-10de4c5 2020-07-17T14:24:25.018 INFO:teuthology.results:seed: '2706' 2020-07-17T14:30:13.485 INFO:root:teuthology version: 1.0.0-10de4c5 2020-07-17T14:30:13.490 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-07-18T00:08:27.872 INFO:teuthology.results:Tests finished! gathering results... 2020-07-18T00:08:30.396 INFO:scrape:Found 16 jobs 2020-07-18T00:08:30.402 INFO:scrape:Found 1 distinct failure reasons 2020-07-18T00:08:30.402 INFO:scrape:Failure: "2020-07-17T15:02:35.779376+0000 mon.b (mon.0) 379 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log 2020-07-18T00:08:30.403 INFO:scrape:4 jobs: ['5235278', '5235270', '5235274', '5235281'] 2020-07-18T00:08:30.403 INFO:scrape:suites intersection: ['2-partial-upgrade/firsthalf', '3-thrash/default', '5-finish-upgrade', '6-octopus', 'start}', 'thrashosds-health', 'ubuntu_latest}'] 2020-07-18T00:08:30.404 INFO:scrape:suites union: ['1-ceph-install/nautilus', '1-nautilus-install/nautilus', '1.1-pg-log-overrides/normal_pg_log', '1.1-pg-log-overrides/short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '3.1-objectstore/bluestore-bitmap', '3.1-objectstore/filestore-xfs', '4-ec-workload/{rados-ec-workload', '4-workload/{radosbench', '5-finish-upgrade', '6-octopus', '7-final-workload', '7-msgr2', '8-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'rbd-cls', 'rbd-ec-workload}', 'rbd-import-export', 'rbd_api', 'readwrite', 'rgw_ragweed_prepare', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}', 'upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack', 'upgrade:nautilus-x/stress-split/{0-cluster/{openstack'] 2020-07-18T00:08:30.404 INFO:scrape: 2020-07-18T00:08:30.406 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 12 Fail ================================================================= [5235270] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 02:15:55 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235270/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235270/ "2020-07-17T15:09:25.361536+0000 mon.c (mon.0) 262 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log [5235274] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:43:36 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235274/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235274/ "2020-07-17T23:00:59.687757+0000 mon.c (mon.0) 282 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log [5235278] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 01:59:17 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235278/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235278/ "2020-07-17T15:02:35.779376+0000 mon.b (mon.0) 379 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log [5235281] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} ----------------------------------------------------------------- time: 03:21:06 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235281/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235281/ "2020-07-17T16:39:26.520303+0000 mon.a (mon.0) 382 : cluster [ERR] Health check failed: 4 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log Pass ================================================================= [5235268] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:44:24 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235268/ [5235269] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:32:38 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235269/ [5235271] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:38:09 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235271/ [5235272] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:42:34 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235272/ [5235273] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs thrashosds-health ubuntu_latest} time: 03:13:03 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235273/ [5235275] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:39:34 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235275/ [5235276] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:43:29 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235276/ [5235277] upgrade:nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap thrashosds-health ubuntu_latest} time: 03:49:56 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235277/ [5235279] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:39:26 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235279/ [5235280] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/bluestore-bitmap ubuntu_latest} time: 02:44:12 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235280/ [5235282] upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_latest} time: 02:00:58 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235282/ [5235283] upgrade:nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} objectstore/filestore-xfs ubuntu_latest} time: 02:38:20 info: http://pulpito.ceph.com/teuthology-2020-07-17_14:23:04-upgrade:nautilus-x-octopus-distro-basic-smithi/5235283/