Found 100 jobs Found 3 distinct failure reasons Failure: Command failed (workunit test rados/test.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e338e1f7d2e42256305bc55fe03d20b6f3b402e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['6727572'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}'] Dead: b'2022-03-09T09:18:44.885 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6727547'] suites: ['0-kubeadm', '0-nvme-loop', '1-rook', '2-workload/none', '3-final', 'cluster/1-node', 'k8s/1.21', 'net/calico', 'rados/rook/smoke/{0-distro/ubuntu_20.04', 'rook/1.7.2}'] Failure: "2022-03-09T03:36:32.545920+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log 1 jobs: ['6727560'] suites: ['2-inject-leak/mon', 'centos_latest}', 'rados/valgrind-leaks/{1-start']