Found 277 jobs Found 11 distinct failure reasons Failure: "2026-02-26T02:35:20.638614+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 84955.138731 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.0. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 9 jobs: ['71007', '70887', '70972', '71086', '71033', '71047', '70929', '71125', '70894'] suites intersection: [] suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/quincy', '1-install/reef', '1-tasks', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'overrides/ignorelist_health', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] Failure: Command failed (workunit test mon/mon-cluster-log.sh) on trial046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-cluster-log.sh' 1 jobs: ['71057'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] Failure: Command failed (workunit test post-file.sh) on trial011 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['71054', '70918'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] Failure: 2 jobs: ['71097', '71132'] suites intersection: ['rados', 'supported-random-distro$/{centos_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/backfill-toofull', 'rados/singleton/{all/ec-inconsistent-hinfo', 'supported-random-distro$/{centos_latest}}'] Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh' 1 jobs: ['71082'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd-backfill}'] Failure: Command failed on trial130 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_2' 2 jobs: ['70960', '71100'] suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/e2e}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/e2e}'] Failure: "2026-02-26T02:35:53.024248+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)" in cluster log 9 jobs: ['70984', '71091', '70967', '70895', '71075', '70908', '70950', '70913', '71027'] suites intersection: [] suites union: ['1-start', '2-inject-leak/mon', '2-inject-leak/none', '2-inject-leak/osd', '2-services/basic', '3-final}', 'centos_latest}', 'ceph', 'clusters/fixed-4', 'd-thrash/none', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream}', 'rados/singleton/{all/mon-stretch-pool', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'supported-random-distro$/{centos_latest}}', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'validater/valgrind}'] Failure: Command failed (workunit test cephtool/test.sh) on trial085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 4 jobs: ['71017', '70883', '70952', '71083'] suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] Failure: Command failed on trial016 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5' 1 jobs: ['70988'] suites: ['clusters/{fixed-2', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/basic/{ceph', 'supported-random-distro$/{centos_latest}', 'tasks/rados_workunit_loadgen_big}'] Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial061 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' ceph version 19.2.3-687-g8d724284 (8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fabdb878520] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x564cfe023094] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x564cfe0229c9] 4: (BlueStore::_close_db()+0x31c) [0x564cfde74f0c] 5: (BlueStore::umount()+0x239) [0x564cfde7f4d9] 6: ceph-kvstore-tool(+0x1f8ef3) [0x564cfde20ef3] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fabdb85fd90] 9: __libc_start_main() 10: _start() 2 jobs: ['70910', '71045'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8d724284d6d767af9ab8bf4bc6bd4e45f8502cb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 1 jobs: ['71112'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd}']