Found 92 jobs Found 10 distinct failure reasons Failure: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on trial124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 1 jobs: ['55738'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd}'] Failure: "2026-02-18T07:18:51.546996+0000 mon.g (mon.6) 53 : cluster [WRN] Health check failed: 3 network partitions detected (MON_NETSPLIT)" in cluster log 8 jobs: ['55659', '55717', '55652', '55670', '55665', '55662', '55648', '55653'] suites intersection: [] suites union: ['2-inject-leak/mon', '2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/fixed-2', 'clusters/fixed-4', 'd-thrash/default/{default', 'd-thrash/none', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/mon-stretch-pool', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}'] Failure: Command failed on trial129 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_2' 2 jobs: ['55726', '55661'] suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'tasks/e2e}'] Failure: "2026-02-18T07:00:12.215409+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 94022.070490 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.0. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 9 jobs: ['55650', '55664', '55651', '55666', '55712', '55676', '55739', '55672', '55656'] suites intersection: [] suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/quincy', '1-install/reef', '1-tasks', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'overrides/ignorelist_health', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial048 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0bbee73520] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x560d20887324] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x519) [0x560d20886c59] 4: (BlueStore::_close_db()+0x31c) [0x560d206d8b8c] 5: (BlueStore::umount()+0x239) [0x560d206e37b9] 6: ceph-kvstore-tool(+0x1f8f13) [0x560d20684f13] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f0bbee5ad90] 9: __libc_start_main() 10: _start() 2 jobs: ['55654', '55674'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test cephtool/test.sh) on trial082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 4 jobs: ['55709', '55668', '55649', '55660'] suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh' 1 jobs: ['55708'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd-backfill}'] Failure: Command failed (workunit test post-file.sh) on trial020 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['55680', '55655'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: 1 jobs: ['55723'] suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test mon/mon-cluster-log.sh) on trial171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-cluster-log.sh' 1 jobs: ['55683'] suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/mon}']