Found 263 jobs Found 3 distinct failure reasons Failure: "1664979417.2970104 osd.6 (osd.6) 3 : cluster [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'cache' : 1 ])" in cluster log 1 jobs: ['7054672'] suites: ['clusters/{fixed-3', 'features/readbalance', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/ec-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{rhel_8}', 'workloads/qemu_bonnie}'] Failure: "1664983255.310846 mon.a (mon.0) 198 : cluster [WRN] Health check failed: Degraded data redundancy: 2/3246 objects degraded (0.062%), 1 pg degraded (PG_DEGRADED)" in cluster log 4 jobs: ['7054712', '7054628', '7054690', '7054746'] suites intersection: ['msgr-failures/few', 'openstack}'] suites union: ['clients/mirror', 'cluster/{2-node', 'clusters/{fixed-3', 'config/copy-on-read', 'config/permit-partial-discard', 'min-compat-client/default', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'pool/replicated-data-pool', 'pool/small-cache-pool', 'rbd-mirror/four-per-cluster', 'rbd/librbd/{cache/writeback', 'rbd/librbd/{cache/writethrough', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_defaults}', 'workloads/c_api_tests_with_journaling}', 'workloads/rbd-mirror-fsx-workunit}'] Dead: b'2022-10-05T23:06:52.812 DEBUG:teuthology.exit:Finished running handlers' ceph version 18.0.0-285-g3288199c (3288199c5f217002f2818832a420ae955d3dfdf8) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f2b49239420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x55867e49cd5a] 5: (KernelDevice::_aio_thread()+0xe41) [0x55867f0b4551] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55867f0b9ab1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f2b4922d609] 8: clone() 1 jobs: ['7054542'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}']