Found 181 jobs Found 6 distinct failure reasons Failure: Command failed on smithi099 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 2 jobs: ['7426002', '7425961'] suites intersection: ['2-cluster/{fix-2', '4-cache-path', '7-workloads/qemu_xfstests', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-cache-size/1G', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'conf/{disable-pool-app}}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] Failure: "1697324582.966496 osd.0 (osd.0) 3 : cluster [ERR] osd.0 found snap mapper error on pg 1.6 oid 1:60bee5c0:::rbd_data.127b3addcd9f.000000000000000c:49 snaps missing in mapper, should be: {45, 46, 47, 48, 49} ...repaired" in cluster log 1 jobs: ['7426006'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 1 jobs: ['7426017'] suites: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/layering', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd_cli_generic}'] Failure: Command failed (workunit test rbd/cli_generic.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/cli_generic.sh' 1 jobs: ['7425904'] suites: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/format-1', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd_cli_generic}'] Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh 1 jobs: ['7425956'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{rhel_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' ceph version 17.2.6-1415-g9d6d32bb (9d6d32bb3452d5179bde2ee1cfa05df7a65f4586) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55bab8961ea7] 2: (KernelDevice::_aio_thread()+0x1699) [0x55bab961c4e9] 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55bab96202d1] 4: /lib64/libpthread.so.0(+0x81ca) [0x7fb854a361ca] 1 jobs: ['7426022'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}']