Found 221 jobs Missing teuthology log /home/teuthworker/archive/teuthology-2024-03-26_21:16:02-rbd-squid-distro-default-smithi/7624266/teuthology.log Found 5 distinct failure reasons Failure: Command failed (workunit test rbd/rbd_mirror_fsx_compare.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cc7ac7587f2280dab48a00005636a78ca2feb4d3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_fsx_compare.sh' 1 jobs: ['7624315'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-fsx-workunit}'] Dead: None 1 jobs: ['7624266'] Timeout 120 running ceph --cluster cluster2 pg dump --format=json 1 jobs: ['7624292'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] Failure: Command failed on smithi086 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2 jobs: ['7624385', '7624275'] suites intersection: ['2-cluster/{fix-2', '4-cache-path', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{centos_latest}', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] Failure: "2024-03-27T02:18:40.719103+0000 mon.a (mon.0) 209 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log 1 jobs: ['7624194'] suites: ['base/install', 'cluster/{fixed-3', 'conf/{disable-pool-app}', 'openstack}', 'rbd/iscsi/{0-single-container-host', 'workloads/cephadm_iscsi}']