Found 9 jobs Misunderstood line: Misunderstood line: Misunderstood line: Misunderstood line: Misunderstood line: Misunderstood line: Found 4 distinct failure reasons Valgrind: mon (Leak_StillReachable) 3 jobs: ['7576291', '7576287', '7576290'] suites intersection: [] suites union: ['2-inject-leak/none', 'centos_latest}', 'ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'validater/valgrind}'] Crash: "2024-02-29T00:37:44.354537+0000 osd.3 (osd.3) 12 : cluster [WRN] osd.3 ep: 331 scrubber::ReplicaReservations pg[2.5]: timeout on replica reservations (since 2024-02-29 00:37:39)" in cluster log ceph version 18.2.1-638-g055f1d8d (055f1d8d2e5cbd7e37bfd1e7bd8c27a3b6e775e2) reef (stable) 1: /lib64/libc.so.6(+0x54db0) [0x549fdb0] 2: /lib64/libc.so.6(+0xa154c) [0x54ec54c] 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point > >)+0x226) [0xb9b2d6] 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration >, std::chrono::duration >)+0x70) [0xb9b3d0] 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x248) [0xbab998] 6: ceph-osd(+0xaa3f14) [0xbabf14] 7: /lib64/libc.so.6(+0x9f802) [0x54ea802] 1 jobs: ['7576292'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'validater/valgrind}'] Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2 jobs: ['7576288', '7576295'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'validater/valgrind}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=055f1d8d2e5cbd7e37bfd1e7bd8c27a3b6e775e2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2 jobs: ['7576294', '7576289'] suites intersection: ['task/test_cephadm}'] suites union: ['agent/off', 'agent/on', 'mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools', 'rados/cephadm/workunits/{0-distro/ubuntu_20.04', 'task/test_cephadm}']