2021-03-18T23:24:42.518 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-18T23:24:42.521 INFO:teuthology.results:subset: '0/8' 2021-03-18T23:24:42.521 INFO:teuthology.results:seed: '9973' 2021-03-18T23:31:25.698 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-18T23:31:25.737 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-19T01:57:36.522 INFO:teuthology.results:Tests finished! gathering results... 2021-03-19T01:57:42.626 INFO:scrape:Found 14 jobs 2021-03-19T01:57:45.515 INFO:scrape:Found 2 distinct failure reasons 2021-03-19T01:57:45.516 INFO:scrape:Dead: b'2021-03-19T01:54:38.722 DEBUG:teuthology.exit:Finished running handlers' 2021-03-19T01:57:45.517 INFO:scrape:10 jobs: ['5978633', '5978635', '5978639', '5978641', '5978628', '5978634', '5978630', '5978636', '5978638', '5978631'] 2021-03-19T01:57:45.517 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados:basic/{ceph', 'tasks/libcephsqlite}'] 2021-03-19T01:57:45.518 INFO:scrape:suites union: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:basic/{ceph', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/libcephsqlite}'] 2021-03-19T01:57:45.518 INFO:scrape: 2021-03-19T01:57:45.519 INFO:scrape:Failure: "2021-03-18T23:44:31.852666+0000 mon.a (mon.0) 104 : cluster [ERR] Health check failed: no active mgr (MGR_DOWN)" in cluster log 2021-03-19T01:57:45.519 INFO:scrape:4 jobs: ['5978629', '5978632', '5978637', '5978640'] 2021-03-19T01:57:45.520 INFO:scrape:suites intersection: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'openstack}', 'rados', 'rados:basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/libcephsqlite}'] 2021-03-19T01:57:45.522 INFO:scrape:suites union: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados:basic/{ceph', 'supported-random-distro$/{ubuntu_latest}', 'tasks/libcephsqlite}'] 2021-03-19T01:57:45.522 INFO:scrape: 2021-03-19T01:57:45.524 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/ failed: 4 dead: 10 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5978629] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:10:43 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978629/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63d83c097f9b4618a2bb18b1c882ca1f "2021-03-18T23:44:31.852666+0000 mon.a (mon.0) 104 : cluster [ERR] Health check failed: no active mgr (MGR_DOWN)" in cluster log [5978632] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:10:49 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978632/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978632/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ede16612fed4c91a59c9bc0ad2eca14 "2021-03-18T23:44:20.645154+0000 mon.a (mon.0) 107 : cluster [ERR] Health check failed: no active mgr (MGR_DOWN)" in cluster log [5978637] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:10:49 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978637/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978637/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ee4b81b8d0f4a77921720988aebb7ac "2021-03-18T23:47:54.455274+0000 mon.a (mon.0) 105 : cluster [ERR] Health check failed: no active mgr (MGR_DOWN)" in cluster log [5978640] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:10:18 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978640/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978640/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af4fd4c8759a46c991daf45f4a38f968 "2021-03-18T23:49:09.211281+0000 mon.a (mon.0) 105 : cluster [ERR] Health check failed: no active mgr (MGR_DOWN)" in cluster log Dead ================================================================= [5978628] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978628/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978628/ [5978630] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978630/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978630/ [5978631] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978631/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978631/ [5978633] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978633/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978633/ [5978634] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978634/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978634/ [5978635] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978635/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978635/ [5978636] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978636/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978636/ [5978638] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978638/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978638/ [5978639] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978639/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978639/ [5978641] rados:basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/libcephsqlite} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978641/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2021-03-18_23:24:07-rados:basic-wip-pdonnell-testing-20210318.205159-distro-basic-smithi/5978641/