Found 3 jobs Found 2 distinct failure reasons Crash: Command failed on smithi023 with status 6: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph tell osd.1 injectargs -- --filestore_debug_random_read_err=0.0'" ceph version 16.0.0-5625-g1d421102 (1d421102af66c91b42b4c769c7a88260d1a0efc3) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x56061ea61c58] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x56061ea61de2] 6: (ReplicatedBackend::prepare_pull(eversion_t, hobject_t const&, std::shared_ptr, ReplicatedBackend::RPGHandle*)+0x2cc) [0x56061ee485bc] 7: (ReplicatedBackend::recover_object(hobject_t const&, eversion_t, std::shared_ptr, std::shared_ptr, PGBackend::RecoveryHandle*)+0xd8) [0x56061ee55f78] 8: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x1d1) [0x56061ec11a81] 9: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xb10) [0x56061ec65520] 10: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x260) [0x56061ec70f00] 11: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x249) [0x56061eae5db9] 12: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x19) [0x56061ed510e9] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x90c) [0x56061eb03fac] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x56061f14de2c] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56061f1510e0] 16: () 17: clone() 2 jobs: ['5452429', '5452431'] suites intersection: ['msgr-failures/many', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['msgr-failures/many', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'rados', 'rados/singleton/{all/random-eio', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test mon/mon-handle-forward.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1d421102af66c91b42b4c769c7a88260d1a0efc3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-handle-forward.sh' 1 jobs: ['5452430'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}']