Found 181 jobs Found 5 distinct failure reasons Failure: Command failed on smithi133 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 3 jobs: ['7492295', '7492386', '7492315'] suites intersection: ['openstack}'] suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{rhel_8}', '3-supported-random-distro$/{ubuntu_latest}', '4-cache-path', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-cache-size/5G', '7-workloads/qemu_xfstests', 'clusters/{fixed-3', 'conf/{disable-pool-app}', 'conf/{disable-pool-app}}', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'pool/ec-cache-pool', 'rbd/encryption/{cache/writeback', 'rbd/pwl-cache/tmpfs/{1-base/install', 'supported-random-distro$/{rhel_8}', 'workloads/qemu_xfstests_luks1}'] Dead: b'2023-12-15T21:30:07.365 DEBUG:teuthology.exit:Finished running handlers' ceph version 17.2.7-128-g3af8de9f (3af8de9f7fd998cabb0ad31dc02733d37b33a399) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x562947622efb] 2: (KernelDevice::_aio_thread()+0x1699) [0x5629482dd719] 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5629482e1501] 4: /lib64/libpthread.so.0(+0x81cf) [0x7fbcd6f591cf] 1 jobs: ['7492406'] suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 2 jobs: ['7492288', '7492401'] suites intersection: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'msgr-failures/few', 'openstack}', 'pool/small-cache-pool', 'workloads/rbd_cli_generic}'] suites union: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/format-1', 'features/layering', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_cli_generic}'] Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh 1 jobs: ['7492340'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{rhel_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] Failure: "1702627214.8587828 osd.5 (osd.5) 4 : cluster [ERR] osd.5 found snap mapper error on pg 2.4 oid 2:200541ec:::rbd_data.129ab1fc7f16.00000000000000e4:19 snaps in mapper: {}, oi: {17, 18, 19} ...repaired" in cluster log 1 jobs: ['7492290'] suites: ['cluster/{fixed-3', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}']