Found 298 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f8403c8d520] , 2: pthread_kill() Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 27 distinct failure reasons Failure: "2026-02-23T08:22:16.428360+0000 mgr.y (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm" in cluster log 46 jobs suites intersection: [] suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/tentacle', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-ceph', '2-inject-leak/mon', '2-inject-leak/none', '2-inject-leak/osd', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-mgrmodules', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', '4-units/crash}}', '4-units/insights}}', '4-units/module_selftest}}', '4-units/per_module_finisher_stats}}', '4-units/progress}}', 'agent/on', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'base', 'ceph', 'clusters/6', 'clusters/fixed-2', 'clusters/fixed-4', 'clusters/{fixed-2}', 'clusters/{fixed-4}', 'clusters/{three-plus-one}', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-balancer/upmap-read', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'distro/{rpm_latest}', 'ec_optimizations/ec_optimizations_off', 'ec_optimizations/ec_optimizations_on', 'fast/fast', 'fast/normal', 'mem$/{low}', 'mem$/{normal-1}', 'mem$/{normal-2}', 'mgr_ttl_cache/disable', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/{bluestore/{alloc$/{avl}', 'objectstore/{bluestore/{alloc$/{bitmap}', 'objectstore/{bluestore/{alloc$/{btree}', 'objectstore/{bluestore/{alloc$/{hybrid}', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{1M}', 'onode-segment$/{256K}', 'onode-segment$/{512K-onoff}', 'onode-segment$/{512K}', 'onode-segment$/{none}', 'rados', 'rados/basic/{ceph', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/multimon/{ceph', 'rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/valgrind-leaks/{1-start', 'rados/verify/{ceph', 'random-objectstore$/{bluestore/{alloc$/{bitmap}', 'random-objectstore$/{bluestore/{alloc$/{btree}', 'random-objectstore$/{bluestore/{alloc$/{hybrid}', 'random}}}}', 'read-affinity/balance', 'read-affinity/default', 'read-affinity/local', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'rpm_latest', 'rpm_latest}', 'supported-random-distro$/{rpm_latest}', 'task/test_ca_signed_key}', 'task/test_rgw_multisite}', 'tasks/libcephsqlite}', 'tasks/mon_clock_no_skews}', 'tasks/mon_recovery', 'tasks/rados_api_tests', 'tasks/rados_cls_all}', 'tasks/rados_workunit_loadgen_mix}', 'tasks/{1-install', 'thrashers/careful', 'thrashers/careful_host', 'thrashers/default', 'thrashers/default_host', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/mapgap_host', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashers/pggrow_host', 'thrashosds-health', 'thrashosds-health}', 'v1}}}}', 'v2}}}}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=clay-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=isa-k=4-m=2}', 'workloads/ec-rados-plugin=isa-k=6-m=3-technique=cauchy}', 'workloads/ec-rados-plugin=isa-k=6-m=3}', 'workloads/ec-rados-plugin=isa-k=8-m=6-technique=cauchy}', 'workloads/ec-rados-plugin=jerasure-k=2-m=2-crush}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-fast-read}', 'workloads/ec-snaps-few-objects-overwrites}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/redirect_promote_tests}', 'workloads/set-chunks-read}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}', 'write$/{random/{compr$/{no$/{no}}', 'write$/{random/{compr$/{yes$/{lz4}}', 'write$/{random/{compr$/{yes$/{snappy}}', 'write$/{random/{compr$/{yes$/{zstd}}', 'write$/{v1/{compr$/{no$/{no}}', 'write$/{v1/{compr$/{yes$/{lz4}}', 'write$/{v1/{compr$/{yes$/{zlib}}', 'write$/{v2/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{yes$/{lz4}}', 'write$/{v2/{compr$/{yes$/{snappy}}', 'write$/{v2/{compr$/{yes$/{zstd}}'] Failure: Command failed (workunit test cephadm/test_cephadm.sh) on trial133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 3 jobs: ['65753', '65617', '65491'] suites intersection: ['agent/off', 'mon_election/classic', 'task/test_cephadm}'] suites union: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/rocky_10', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] Failure: 1 jobs: ['65710'] suites: ['base', 'mem$/{low}', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{1M}', 'rados', 'rados/singleton/{all/backfill-toofull', 'random}}}}', 'supported-random-distro$/{rpm_latest}}', 'write$/{random/{compr$/{no$/{no}}'] Failure: The WatchDog barked due to Daemon ceph.osd.6 has failed 2 jobs: ['65545', '65757'] suites intersection: ['base', 'ceph', 'clusters/{fixed-4}', 'ec_optimizations/ec_optimizations_on', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-partial-recovery}', 'thrashosds-health'] suites union: ['base', 'ceph', 'clusters/{fixed-4}', 'ec_optimizations/ec_optimizations_on', 'mem$/{low}', 'mem$/{normal-2}', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'objectstore/{bluestore/{alloc$/{btree}', 'objectstore/{bluestore/{alloc$/{hybrid}', 'onode-segment$/{512K-onoff}', 'onode-segment$/{512K}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{rpm_latest}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful_host', 'thrashers/pggrow_host', 'thrashosds-health', 'v1}}}}', 'v2}}}}', 'workloads/ec-rados-plugin=isa-k=10-m=4}', 'workloads/ec-rados-plugin=isa-k=8-m=6-technique=cauchy}', 'write$/{v1/{compr$/{yes$/{zlib}}', 'write$/{v2/{compr$/{no$/{no}}'] Failure: Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial017 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh' 2 jobs: ['65530', '65601'] suites intersection: ['base', 'objectstore/bluestore/{alloc$/{stupid}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['base', 'mem$/{normal-1}', 'mem$/{normal-2}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore/{alloc$/{stupid}', 'onode-segment$/{1M}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'random}}}', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}', 'write$/{random/{compr$/{yes$/{zstd}}', 'write$/{v1/{compr$/{no$/{no}}'] Timeout 120 running ceph --cluster ceph pg dump --format=json 1 jobs: ['65604'] suites: ['ceph', 'clusters/{fixed-4}', 'ec_optimizations/ec_optimizations_on', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/minsize_recovery', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}'] Failure: Command failed on trial019 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:5adce3ff0fe1645dcbc38d059f948571dc3d69da shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 97b6ee52-108d-11f1-a77e-d404e6e7d460 -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3 mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3)+0x102) [0x55f4d4819662] 8: (BlueFS::unlink(std::basic_string_view >, std::basic_string_view >)+0x313) [0x55f4d4839843] 9: (BlueRocksEnv::DeleteFile(std::__cxx11::basic_string, std::allocator > const&)+0x47) [0x55f4d496c7d7] 10: (rocksdb::DeleteDBFile(rocksdb::ImmutableDBOptions const*, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0xb3) [0x55f4d4b1f923] 11: (rocksdb::DBImpl::DeleteObsoleteFileImpl(int, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::FileType, unsigned long)+0xc3) [0x55f4d4a17703] 12: (rocksdb::DBImpl::PurgeObsoleteFiles(rocksdb::JobContext&, bool)+0x1a4e) [0x55f4d4a1cb3e] 13: (rocksdb::DBImpl::DeleteObsoleteFiles()+0xa9) [0x55f4d4a1d429] 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x812) [0x55f4d4a2b4e2] 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x28) [0x55f4d4a2ca48] 16: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0xdf4) [0x55f4d47df744] 17: (BlueStore::_open_db(bool, bool, bool)+0x4e2) [0x55f4d488d1c2] 18: (BlueStore::_open_db_and_around(bool, bool)+0x454) [0x55f4d488e084] 19: (BlueStore::expand_devices(std::ostream&)+0x97d) [0x55f4d48978fd] 20: main() 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f59d554bd90] 22: __libc_start_main() 23: _start() 1 jobs: ['65505'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd}'] Failure: Command failed (workunit test post-file.sh) on trial194 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['65591', '65742'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on trial119 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:5adce3ff0fe1645dcbc38d059f948571dc3d69da shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5e13131d-1093-11f1-aa61-d404e6e7d460 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n# check prometheus metrics endpoint is not empty and make sure we can get metrics\nMETRICS_URL=$(ceph mgr services | jq -r .prometheus)\n[ -n "$METRICS_URL" ] || exit 1\ncurl -s "${METRICS_URL}metrics" | grep -q \'"\'"\'^ceph_health_status\'"\'"\'\n\'' 2 jobs: ['65697', '65560'] suites intersection: ['agent/off', 'mon_election/classic', 'task/test_monitoring_stack_basic}'] suites union: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'rados/cephadm/workunits/{0-distro/rocky_10', 'task/test_monitoring_stack_basic}'] Timeout 3h running clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 1 jobs: ['65685'] suites: ['rados/standalone/{supported-random-distro$/{rpm_latest}', 'workloads/misc}'] Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x196) [0x7f8404353f26] 1 jobs: ['65745'] suites: ['rados/objectstore/{backends/objectstore-bluestore-a', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test mon/mon-stretch-mode-5-mons-8-osds.sh) on trial173 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/mon-stretch-mode-5-mons-8-osds.sh' 1 jobs: ['65647'] suites: ['base', 'mem$/{normal-1}', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{btree}', 'onode-segment$/{1M}', 'rados', 'rados/singleton/{all/stretch-mode-5-mons-8-osds', 'random}}}}', 'supported-random-distro$/{ubuntu_latest}}', 'write$/{random/{compr$/{yes$/{snappy}}'] Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial075 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - None) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f6fe1255520] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xc7) [0x55e067ed52b7] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x53f) [0x55e067ece82f] 4: (BlueStore::_close_db()+0x324) [0x55e067d33964] 5: (BlueStore::umount()+0x195) [0x55e067d3c6d5] 6: ceph-kvstore-tool(+0x20e293) [0x55e067cdb293] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f6fe123cd90] 9: __libc_start_main() 10: _start() 2 jobs: ['65735', '65584'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed (workunit test mon/misc.sh) on trial072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' 1 jobs: ['65746'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] Crash: Command failed (workunit test rados/test_crash.sh) on trial050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh' ceph version 20.3.0-5433-g5adce3ff (5adce3ff0fe1645dcbc38d059f948571dc3d69da) tentacle (dev - RelWithDebInfo) 1: /lib64/libc.so.6(+0x41000) [0x7fd724069000] 2: /lib64/libc.so.6(+0x920ae) [0x7fd7240ba0ae] 3: pthread_cond_wait() 4: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fd7242e3da0] 5: (AsyncMessenger::wait()+0x6f) [0x55ecbff2d03f] 6: main() 7: /lib64/libc.so.6(+0x2a58e) [0x7fd72405258e] 8: __libc_start_main() 9: _start() 1 jobs: ['65656'] suites: ['base', 'mem$/{normal-2}', 'mon_election/classic', 'msgr-failures/none', 'msgr/async', 'objectstore/{bluestore/{alloc$/{hybrid}', 'onode-segment$/{256K}', 'rados', 'rados/singleton/{all/test-crash', 'supported-random-distro$/{rpm_latest}}', 'v2}}}}', 'write$/{v2/{compr$/{yes$/{zlib}}'] Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=rocky%2F10%2Fx86_64&ref=reef 2 jobs: ['65562', '65717'] suites intersection: ['1-tasks'] suites union: ['0-start', '1-tasks', 'mon_election/connectivity', 'overrides/ignorelist_health', 'rados/encoder/{0-start', 'rados/upgrade/parallel/{0-random-distro$/{rocky_10}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'supported-random-distro$/{rpm_latest}}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Failure: Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus.test_standby) 2 jobs: ['65744', '65521'] suites intersection: ['2-ceph', '3-mgrmodules', '4-units/prometheus}}', 'base', 'debug/mgr', 'rados/mgr/{clusters/{2-node-mgr}', 'tasks/{1-install'] suites union: ['2-ceph', '3-mgrmodules', '4-units/prometheus}}', 'base', 'debug/mgr', 'distro/{rpm_latest}', 'distro/{ubuntu_latest}', 'mem$/{normal-1}', 'mem$/{normal-2}', 'mgr_ttl_cache/disable', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'onode-segment$/{256K}', 'onode-segment$/{none}', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore/{alloc$/{bitmap}', 'random-objectstore$/{bluestore/{alloc$/{stupid}', 'random}}}}', 'tasks/{1-install', 'v2}}}}', 'write$/{random/{compr$/{no$/{no}}', 'write$/{v2/{compr$/{no$/{no}}'] Failure: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_enabled_module) 2 jobs: ['65561', '65713'] suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/dashboard}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/dashboard}'] Failure: Command failed (workunit test cephtool/test.sh) on trial129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 2 jobs: ['65731', '65665'] suites intersection: ['base', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{rpm_latest}}'] suites union: ['base', 'mem$/{low}', 'mem$/{normal-2}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore/{alloc$/{avl}', 'objectstore/bluestore/{alloc$/{hybrid}', 'onode-segment$/{512K}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'random}}}', 'supported-random-distro$/{rpm_latest}}', 'v2}}}', 'write$/{random/{compr$/{yes$/{snappy}}', 'write$/{v2/{compr$/{no$/{no}}'] Failure: Command failed (workunit test cephadm/test_repos.sh) on trial155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' 1 jobs: ['65764'] suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm_repos}'] Failure: Command failed on trial169 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5433-g5adce3ff-1jammy cephadm=20.3.0-5433-g5adce3ff-1jammy ceph-mds=20.3.0-5433-g5adce3ff-1jammy ceph-mgr=20.3.0-5433-g5adce3ff-1jammy ceph-osd=20.3.0-5433-g5adce3ff-1jammy ceph-osd-classic=20.3.0-5433-g5adce3ff-1jammy ceph-common=20.3.0-5433-g5adce3ff-1jammy ceph-fuse=20.3.0-5433-g5adce3ff-1jammy ceph-test=20.3.0-5433-g5adce3ff-1jammy ceph-volume=20.3.0-5433-g5adce3ff-1jammy radosgw=20.3.0-5433-g5adce3ff-1jammy python3-rados=20.3.0-5433-g5adce3ff-1jammy python3-rgw=20.3.0-5433-g5adce3ff-1jammy python3-cephfs=20.3.0-5433-g5adce3ff-1jammy python3-rbd=20.3.0-5433-g5adce3ff-1jammy libcephfs2=20.3.0-5433-g5adce3ff-1jammy libcephfs-dev=20.3.0-5433-g5adce3ff-1jammy librados2=20.3.0-5433-g5adce3ff-1jammy librbd1=20.3.0-5433-g5adce3ff-1jammy rbd-fuse=20.3.0-5433-g5adce3ff-1jammy' 1 jobs: ['65574'] suites: ['base', 'ceph', 'clusters/{fixed-4}', 'ec_optimizations/ec_optimizations_off', 'mem$/{low}', 'mon_election/classic', 'msgr-failures/fastclose', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{512K-onoff}', 'rados', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'v1}}}}', 'workloads/ec-rados-plugin=isa-k=10-m=4}', 'write$/{v1/{compr$/{yes$/{lz4}}'] Dead: b'2026-02-23T14:14:41.906 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['65642'] suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'base', 'ceph', 'clusters/{fixed-4}', 'crc-failures/default', 'd-balancer/crush-compat', 'mem$/{normal-1}', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{hybrid}', 'onode-segment$/{none}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'v2}}}}', 'workloads/pool-snaps-few-objects}', 'write$/{v2/{compr$/{yes$/{lz4}}'] Failure: Command failed (workunit test rados/test.sh) on trial096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['65683'] suites: ['base', 'clusters/9-mons', 'mem$/{normal-2}', 'mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{512K-onoff}', 'rados', 'rados/monthrash/{ceph', 'random}}}}', 'supported-random-distro$/{rpm_latest}', 'thrashers/one', 'workloads/rados_api_tests}', 'write$/{random/{compr$/{no$/{no}}'] Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5adce3ff0fe1645dcbc38d059f948571dc3d69da TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh' 1 jobs: ['65479'] suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd-backfill}']