2021-03-22T07:02:28.641 INFO:root:teuthology version: 1.1.0-6b3150e9 2021-03-22T07:02:28.642 INFO:teuthology.results:subset: '6025/9999' 2021-03-22T07:02:28.642 INFO:teuthology.results:seed: '6476' 2021-03-22T07:30:38.847 INFO:root:teuthology version: 1.1.0-6b3150e9 2021-03-22T07:30:38.889 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-22T09:17:46.449 INFO:teuthology.results:Tests finished! gathering results... 2021-03-22T09:17:47.546 INFO:scrape:Found 15 jobs 2021-03-22T09:17:47.548 INFO:scrape:Found 2 distinct failure reasons 2021-03-22T09:17:47.548 INFO:scrape:Failure: Command failed on smithi205 with status 123: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2021-03-22T09:17:47.548 INFO:scrape:1 jobs: ['5985991'] 2021-03-22T09:17:47.548 INFO:scrape:suites: ['objectstore/filestore-xfs', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/cfuse_workunit_misc', 'thrashosds-health', 'whitelist_health}'] 2021-03-22T09:17:47.548 INFO:scrape: 2021-03-22T09:17:47.548 INFO:scrape:Failure: "2021-03-22T07:32:49.735685+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2021-03-22T09:17:47.549 INFO:scrape:1 jobs: ['5985985'] 2021-03-22T09:17:47.549 INFO:scrape:suites: ['objectstore/bluestore-comp-snappy', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-all-distro/ubuntu_latest', 'tasks/radosbench', 'thrashosds-health', 'whitelist_health}'] 2021-03-22T09:17:47.549 INFO:scrape: 2021-03-22T09:17:47.549 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [5985985] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/ubuntu_latest tasks/radosbench thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 01:50:23 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985985/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985985/ "2021-03-22T07:32:49.735685+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log [5985991] powercycle/osd/{clusters/3osd-1per-target objectstore/filestore-xfs powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_misc thrashosds-health whitelist_health} ----------------------------------------------------------------- time: 00:30:02 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985991/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985991/ Command failed on smithi205 with status 123: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" Pass ================================================================= [5985980] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-low-osd-mem-target powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_fsx thrashosds-health whitelist_health} time: 00:16:56 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985980/ [5985981] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-stupid powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_fsync thrashosds-health whitelist_health} time: 00:14:07 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985981/ [5985982] powercycle/osd/{clusters/3osd-1per-target objectstore/filestore-xfs powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_pjd thrashosds-health whitelist_health} time: 00:11:24 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985982/ [5985983] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-bitmap powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_truncate_delay thrashosds-health whitelist_health} time: 00:09:18 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985983/ [5985984] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/rhel_8 tasks/rados_api_tests thrashosds-health whitelist_health} time: 00:27:55 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985984/ [5985986] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zlib powercycle/default supported-all-distro/centos_8 tasks/readwrite thrashosds-health whitelist_health} time: 00:15:23 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985986/ [5985987] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-zstd powercycle/default supported-all-distro/rhel_8 tasks/snaps-few-objects thrashosds-health whitelist_health} time: 00:25:20 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985987/ [5985988] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-hybrid powercycle/default supported-all-distro/ubuntu_latest tasks/snaps-many-objects thrashosds-health whitelist_health} time: 00:23:44 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985988/ [5985989] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-low-osd-mem-target powercycle/default supported-all-distro/centos_8 tasks/admin_socket_objecter_requests thrashosds-health whitelist_health} time: 00:14:44 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985989/ [5985990] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-stupid powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_kernel_untar_build thrashosds-health whitelist_health} time: 00:48:19 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985990/ [5985992] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-bitmap powercycle/default supported-all-distro/centos_8 tasks/cfuse_workunit_suites_ffsb thrashosds-health whitelist_health} time: 00:25:26 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985992/ [5985993] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-lz4 powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_suites_fsstress thrashosds-health whitelist_health} time: 00:13:58 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985993/ [5985994] powercycle/osd/{clusters/3osd-1per-target objectstore/bluestore-comp-snappy powercycle/default supported-all-distro/ubuntu_latest tasks/cfuse_workunit_suites_fsx thrashosds-health whitelist_health} time: 00:24:59 info: http://pulpito.ceph.com/teuthology-2021-03-21_12:17:01-powercycle-pacific-distro-basic-smithi/5985994/