2020-09-16T10:42:32.106 INFO:root:teuthology version: 1.0.0-55bd2dc
2020-09-16T10:42:32.110 INFO:teuthology.results:subset: '8/5000'
2020-09-16T10:42:32.110 INFO:teuthology.results:seed: '804'
2020-09-16T21:57:33.442 INFO:root:teuthology version: 1.0.0-55bd2dc
2020-09-16T21:57:33.525 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2020-09-17T03:11:47.623 INFO:teuthology.results:Tests finished! gathering results...
2020-09-17T03:12:56.661 INFO:scrape:Found 592 jobs
2020-09-17T03:13:03.297 INFO:scrape:Found 5 distinct failure reasons
2020-09-17T03:13:03.300 INFO:scrape:Failure: Command failed on smithi185 with status 6: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600"
2020-09-17T03:13:03.300 INFO:scrape:1 jobs: ['5438905']
2020-09-17T03:13:03.300 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/random', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/redirect}']
2020-09-17T03:13:03.301 INFO:scrape:
2020-09-17T03:13:03.301 INFO:scrape:Failure: Command failed (workunit test cephtool/test.sh) on smithi055 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5a831a88f8077d1a43df0dac22b16a5bc84b6413 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
2020-09-17T03:13:03.301 INFO:scrape:6 jobs: ['5438616', '5438854', '5438718', '5439051', '5438951', '5439149']
2020-09-17T03:13:03.302 INFO:scrape:suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool']
2020-09-17T03:13:03.302 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'msgr/random', 'msgr/simple', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{rhel_latest}}', 'supported-random-distro$/{ubuntu_16.04}}', 'supported-random-distro$/{ubuntu_latest}}']
2020-09-17T03:13:03.302 INFO:scrape:
2020-09-17T03:13:03.303 INFO:scrape:Failure: "2020-09-16 21:38:44.045058 mon.b (mon.0) 10 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log
2020-09-17T03:13:03.303 INFO:scrape:2 jobs: ['5439168', '5438610']
2020-09-17T03:13:03.303 INFO:scrape:suites intersection: ['mon_election/connectivity', 'msgr/async', 'rados']
2020-09-17T03:13:03.304 INFO:scrape:suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/off', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton/{all/mon-config', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_16.04}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/redirect_set_object}']
2020-09-17T03:13:03.304 INFO:scrape:
2020-09-17T03:13:03.305 INFO:scrape:Failure: "2020-09-16 11:24:49.571191 osd.3 (osd.3) 131 : cluster [ERR] 2.18 scrub : stat mismatch, got 11/10 objects, 0/0 clones, 11/10 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 30206106/27365261 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2020-09-17T03:13:03.305 INFO:scrape:1 jobs: ['5438623']
2020-09-17T03:13:03.305 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/upmap', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_16.04}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/write_fadvise_dontneed}']
2020-09-17T03:13:03.306 INFO:scrape:
2020-09-17T03:13:03.306 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2020-09-17T03:13:03.306 INFO:scrape:1 jobs: ['5438875']
2020-09-17T03:13:03.306 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_16.04}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}']
2020-09-17T03:13:03.307 INFO:scrape: