2021-05-19T23:03:04.508 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-19T23:03:04.510 INFO:teuthology.results:seed: '5399' 2021-05-19T23:19:43.853 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-19T23:19:43.857 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-19T23:48:45.852 INFO:teuthology.results:Tests finished! gathering results... 2021-05-19T23:48:47.035 INFO:scrape:Found 18 jobs 2021-05-19T23:48:53.113 INFO:scrape:Found 2 distinct failure reasons 2021-05-19T23:48:53.114 INFO:scrape:Failure: Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) 2021-05-19T23:48:53.114 INFO:scrape:12 jobs: ['6124097', '6124098', '6124088', '6124096', '6124094', '6124091', '6124092', '6124089', '6124102', '6124103', '6124101', '6124099'] 2021-05-19T23:48:53.115 INFO:scrape:suites intersection: ['debug/mgr', 'rados:mgr/{clusters/{2-node-mgr}', 'tasks/progress}'] 2021-05-19T23:48:53.115 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rados:mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/progress}'] 2021-05-19T23:48:53.116 INFO:scrape: 2021-05-19T23:48:53.116 INFO:scrape:Failure: Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) 2021-05-19T23:48:53.117 INFO:scrape:4 jobs: ['6124093', '6124105', '6124100', '6124090'] 2021-05-19T23:48:53.117 INFO:scrape:suites intersection: ['debug/mgr', 'rados:mgr/{clusters/{2-node-mgr}', 'tasks/progress}'] 2021-05-19T23:48:53.118 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rados:mgr/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/progress}'] 2021-05-19T23:48:53.118 INFO:scrape: 2021-05-19T23:48:53.120 INFO:teuthology.results:Sending results to ksirivad@redhat.com: Test Run: ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6124088] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:11:07 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124088/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2e607247a7844cb986370738f879b52 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124089] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:10:52 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124089/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4eb307372e344d47b1468a65fb691350 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124090] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:22:51 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124090/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03039331be454bceb4c2adebd401112f Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) [6124091] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-stupid supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:10:52 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124091/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e96c8e90f7248d7b4cd85d5f0427aa2 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124092] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:09:17 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124092/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124092/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc46abe5934142e5a6efbbffc4b2909e Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124093] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zlib supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:20:16 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124093/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6b3130fe95e4bb98e2e619d0b1c0d6c Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) [6124094] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:09:31 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124094/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124094/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40041a5cad6e4c74afeff85e6badb365 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124096] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-low-osd-mem-target supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:20:05 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124096/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e124317e31a946af8a386e155d91b038 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124097] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:09:42 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124097/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06998c5085594eb391f8f46cd4c9be57 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124098] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124098/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124098/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac7643f654434ebf9de6db405921210f Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124099] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:09:15 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124099/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124099/ sentry: https://sentry.ceph.com/organizations/ceph/?query=182de3723b054761b874135f0e73a34b Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124100] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-stupid supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:20:26 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124100/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1484a1fe4e647daa19f2a765a7b37fd Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) [6124101] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-hybrid supported-random-distro$/{rhel_8} tasks/progress} ----------------------------------------------------------------- time: 00:19:27 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124101/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a8a5f51f7724a6a93d5f3b8ac476aeb Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124102] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:08:56 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124102/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124102/ sentry: https://sentry.ceph.com/organizations/ceph/?query=24d4be27454546f2bc6fd7ca383c6ba4 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124103] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest} tasks/progress} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124103/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c653abe1cc214b55bca58cef8e9bdad3 Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress) [6124105] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:11:55 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124105/ log: http://qa-proxy.ceph.com/teuthology/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba1ce13be0fb44c788cecd92c9345f74 Test failure: test_osd_cannot_recover (tasks.mgr.test_progress.TestProgress) Pass ================================================================= [6124095] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity objectstore/filestore-xfs supported-random-distro$/{centos_8} tasks/progress} time: 00:15:00 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124095/ [6124104] rados:mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic objectstore/filestore-xfs supported-random-distro$/{rhel_8} tasks/progress} time: 00:24:29 info: http://pulpito.ceph.com/ksirivad-2021-05-19_21:54:26-rados:mgr-wip-ksirivad-fix-bug-49988-distro-basic-smithi/6124104/