Found 367 jobs Found 5 distinct failure reasons Failure: SSH connection to smithi121 was lost: 'uname -r' 3 jobs: ['6640709', '6640683', '6640730'] suites intersection: ['rados', 'supported-random-distro$/{ubuntu_18.04}}'] suites union: ['msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton-nomsgr/{all/multi-backfill-reject', 'rados/singleton-nomsgr/{all/osd_stale_reads', 'rados/singleton/{all/divergent_priors', 'supported-random-distro$/{ubuntu_18.04}}'] Failure: 'wait for quorum size 9' reached maximum tries (100) after waiting for 300 seconds 1 jobs: ['6640762'] suites: ['clusters/9-mons', 'msgr-failures/mon-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{ubuntu_18.04}', 'thrashers/force-sync-many', 'workloads/pool-create-delete}'] Failure: Command failed on smithi077 with status 5: 'sudo systemctl stop ceph-345b5f84-7f4c-11ec-8c35-001a4aab830c@mon.a' 4 jobs: ['6640971', '6640853', '6641038', '6640787'] suites intersection: ['3-start-upgrade', '4-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8'] suites union: ['2-repo_digest/defaut', '2-repo_digest/repo_digest', '3-start-upgrade', '4-wait', 'fixed-2}', 'rados/cephadm/upgrade/{1-start-distro/1-start-centos_8'] Failure: Command failed on smithi140 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-filestore 20 --debug-bluestore 20" ceph_test_objectstore --gtest_filter=-*/3 --gtest_catch_exceptions=0\'' 1 jobs: ['6640775'] suites: ['rados/objectstore/{backends/objectstore', 'supported-random-distro$/{centos_8}}'] Failure: wait_for_recovery: failed before timeout expired 1 jobs: ['6640834'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/crush-compat', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/rados_api_tests}']