Found 19 jobs Found 4 distinct failure reasons Failure: "1684431662.839964 mon.a (mon.0) 249 : cluster [WRN] Health check failed: Degraded data redundancy: 1/120 objects degraded (0.833%), 1 pg degraded (PG_DEGRADED)" in cluster log 1 jobs: ['7277724'] suites: ['clusters/{fixed-3', 'config/none', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}'] Failure: Command failed on smithi103 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 1 jobs: ['7277727'] suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_20.04}', '4-cache-path', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests}', 'openstack}', 'rbd/pwl-cache/tmpfs/{1-base/install'] Dead: b"2023-05-19T04:42:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi160.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls" ceph version 18.0.0-3137-gb7effb87 (b7effb87b9c2fcd75d268d929fe6c286fb3c1c08) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7efe42846ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d30e7cfce1] 5: (KernelDevice::_aio_thread()+0x1798) [0x55d30f3dce38] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d30f3df4d1] 7: /lib64/libpthread.so.0(+0x81cf) [0x7efe4283c1cf] 8: clone() 1 jobs: ['7277726'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] Timeout 120 running ceph --cluster ceph pg dump --format=json 1 jobs: ['7277711'] suites: ['clusters/{fixed-3', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/ec-cache-pool', 'rbd/encryption/{cache/none', 'supported-random-distro$/{centos_8}', 'workloads/qemu_xfstests_luks1_luks1}']