2026-02-21T21:56:13.966 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-21T21:56:13.966 INFO:teuthology.results:subset: '111/120000' 2026-02-21T21:56:13.966 INFO:teuthology.results:seed: '1998' 2026-02-21T23:12:13.064 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-21T23:12:13.072 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-22T03:08:32.114 INFO:teuthology.results:Tests finished! gathering results... 2026-02-22T03:08:37.883 INFO:scrape:Found 272 jobs 2026-02-22T03:08:57.580 INFO:scrape:Found 25 distinct failure reasons 2026-02-22T03:08:57.580 INFO:scrape:Failure: Command failed (workunit test cephtool/test_crimson_dump_metrics.sh) on trial124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_crimson_dump_metrics.sh' 2026-02-22T03:08:57.580 INFO:scrape:5 jobs: ['63102', '63281', '63037', '63220', '63160'] 2026-02-22T03:08:57.580 INFO:scrape:suites intersection: ['base', 'rados', 'rados/singleton-bluestore/{all/cephtool'] 2026-02-22T03:08:57.580 INFO:scrape:suites union: ['base', 'mem$/{low}', 'mem$/{normal-1}', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore/{alloc$/{avl}', 'objectstore/bluestore/{alloc$/{bitmap}', 'objectstore/bluestore/{alloc$/{hybrid}', 'onode-segment$/{256K}', 'onode-segment$/{512K-onoff}', 'onode-segment$/{512K}', 'onode-segment$/{none}', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'random}}}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}', 'write$/{random/{compr$/{yes$/{snappy}}', 'write$/{random/{compr$/{yes$/{zlib}}', 'write$/{random/{compr$/{yes$/{zstd}}', 'write$/{v1/{compr$/{yes$/{lz4}}'] 2026-02-22T03:08:57.580 INFO:scrape: 2026-02-22T03:08:57.580 INFO:scrape:Failure: Command failed (workunit test mon/misc.sh) on trial028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/misc.sh' 2026-02-22T03:08:57.580 INFO:scrape:1 jobs: ['63234'] 2026-02-22T03:08:57.580 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/mon}'] 2026-02-22T03:08:57.580 INFO:scrape: 2026-02-22T03:08:57.580 INFO:scrape:Failure: failed to complete snap trimming before timeout 2026-02-22T03:08:57.581 INFO:scrape:1 jobs: ['63138'] 2026-02-22T03:08:57.581 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-5}', 'backoff/normal', 'base', 'ceph', 'clusters/{fixed-4}', 'crc-failures/default', 'd-balancer/crush-compat', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{hybrid}', 'onode-segment$/{256K}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'random}}}}', 'supported-random-distro$/{centos_latest}', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'write$/{random/{compr$/{yes$/{zlib}}'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: Command failed (workunit test cephadm/test_iscsi_setup.sh) on trial165 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_iscsi_setup.sh' 2026-02-22T03:08:57.581 INFO:scrape:1 jobs: ['63159'] 2026-02-22T03:08:57.581 INFO:scrape:suites: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'task/test_iscsi_container/{centos_9.stream', 'test_iscsi_container}}'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: Command failed on trial046 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch reef https://github.com/NitzanMordhai/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0' 2026-02-22T03:08:57.581 INFO:scrape:2 jobs: ['63207', '63072'] 2026-02-22T03:08:57.581 INFO:scrape:suites intersection: ['0-start', '1-tasks', 'overrides/ignorelist_health', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2026-02-22T03:08:57.581 INFO:scrape:suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'overrides/ignorelist_health', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: 2026-02-22T03:08:57.581 INFO:scrape:1 jobs: ['63201'] 2026-02-22T03:08:57.581 INFO:scrape:suites: ['base', 'mem$/{low}', 'mon_election/classic', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{bitmap}', 'onode-segment$/{256K}', 'rados', 'rados/singleton/{all/backfill-toofull', 'supported-random-distro$/{ubuntu_latest}}', 'v1}}}}', 'write$/{v1/{compr$/{yes$/{snappy}}'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: ["Unexpected failure during module execution: [Errno 28] No space left on device: '/home/teuthworker/.ansible/tmp/ansible-local-1588736nnyxqazs/tmp1_tft26u'"] 2026-02-22T03:08:57.581 INFO:scrape:2 jobs: ['63285', '63279'] 2026-02-22T03:08:57.581 INFO:scrape:suites intersection: ['mon_election/connectivity'] 2026-02-22T03:08:57.581 INFO:scrape:suites union: ['base', 'clusters/{fixed-4}', 'ec_optimizations/ec_optimizations_on', 'fast/fast', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/few', 'objectstore/bluestore-basic-min-osd-mem-target', 'objectstore/{bluestore/{alloc$/{btree}', 'onode-segment$/{none}', 'rados', 'rados/perf/{ceph', 'rados/thrash-erasure-code/{ceph', 'random}}}}', 'recovery-overrides/{more-async-partial-recovery}', 'scheduler/dmclock_default_shards', 'settings/optimized', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/fastread', 'thrashosds-health', 'ubuntu_latest', 'workloads/ec-rados-plugin=jerasure-k=3-m=1}', 'workloads/fio_4M_rand_rw}', 'write$/{random/{compr$/{no$/{no}}'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: Test failure: test_disable_stretch_mode (tasks.stretch_mode_disable_enable.TestStretchMode.test_disable_stretch_mode) 2026-02-22T03:08:57.581 INFO:scrape:1 jobs: ['63143'] 2026-02-22T03:08:57.581 INFO:scrape:suites: ['base', 'mem$/{normal-1}', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async-v2only', 'objectstore/{bluestore/{alloc$/{btree}', 'onode-segment$/{256K}', 'rados', 'rados/singleton/{all/stretch-mode-5-mons-8-osds', 'supported-random-distro$/{centos_latest}}', 'v1}}}}', 'write$/{v1/{compr$/{yes$/{zstd}}'] 2026-02-22T03:08:57.581 INFO:scrape: 2026-02-22T03:08:57.581 INFO:scrape:Failure: Command failed (workunit test post-file.sh) on trial139 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2026-02-22T03:08:57.581 INFO:scrape:2 jobs: ['63230', '63092'] 2026-02-22T03:08:57.582 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file'] 2026-02-22T03:08:57.582 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on trial155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2026-02-22T03:08:57.582 INFO:scrape:2 jobs: ['63290', '63055'] 2026-02-22T03:08:57.582 INFO:scrape:suites intersection: ['agent/on', 'mon_election/connectivity', 'task/test_cephadm}'] 2026-02-22T03:08:57.582 INFO:scrape:suites union: ['agent/on', 'mon_election/connectivity', 'rados/cephadm/workunits/{0-distro/centos_9.stream', 'rados/cephadm/workunits/{0-distro/ubuntu_22.04', 'task/test_cephadm}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Failure: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest.test_list_enabled_module) 2026-02-22T03:08:57.582 INFO:scrape:2 jobs: ['63065', '63203'] 2026-02-22T03:08:57.582 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'tasks/dashboard}'] 2026-02-22T03:08:57.582 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-hybrid}', 'tasks/dashboard}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Failure: reached maximum tries (500) after waiting for 3000 seconds 2026-02-22T03:08:57.582 INFO:scrape:1 jobs: ['63179'] 2026-02-22T03:08:57.582 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'base', 'ceph', 'clusters/{fixed-4}', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mem$/{normal-2}', 'mon_election/classic', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{1M}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'random}}}}', 'supported-random-distro$/{centos_latest}', 'thrashers/pggrow_host', 'thrashosds-health', 'workloads/radosbench}', 'write$/{random/{compr$/{no$/{no}}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial150 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' 2026-02-22T03:08:57.582 INFO:scrape:ceph version 20.3.0-5425-g7c8cf961 (7c8cf961761b37084e637a305d42f77c52437802) tentacle (dev - None) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fcb2d67b520] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xc7) [0x55dc4478f2b7] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x53f) [0x55dc4478882f] 4: (BlueStore::_close_db()+0x324) [0x55dc445ed964] 5: (BlueStore::umount()+0x195) [0x55dc445f66d5] 6: ceph-kvstore-tool(+0x20e293) [0x55dc44595293] 7: main() 8: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcb2d662d90] 9: __libc_start_main() 10: _start() 2026-02-22T03:08:57.582 INFO:scrape:2 jobs: ['63086', '63224'] 2026-02-22T03:08:57.582 INFO:scrape:suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] 2026-02-22T03:08:57.582 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{ubuntu_latest}}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Failure: "2026-02-21T22:20:33.242124+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 3 pool(s) nearfull (POOL_NEARFULL)" in cluster log 2026-02-22T03:08:57.582 INFO:scrape:1 jobs: ['63115'] 2026-02-22T03:08:57.582 INFO:scrape:suites: ['mon_election/classic', 'objectstore/bluestore-bitmap', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/radosbench_4M_rand_read}'] 2026-02-22T03:08:57.582 INFO:scrape: 2026-02-22T03:08:57.582 INFO:scrape:Failure: Command failed on trial041 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2' 2026-02-22T03:08:57.582 INFO:scrape:2 jobs: ['63272', '63135'] 2026-02-22T03:08:57.582 INFO:scrape:suites intersection: ['debug/mgr', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/e2e}'] 2026-02-22T03:08:57.583 INFO:scrape:suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'tasks/e2e}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Failure: The WatchDog barked due to Thrasher OSDThrasher threw exception wait_for_clean: failed before timeout expired 2026-02-22T03:08:57.583 INFO:scrape:1 jobs: ['63042'] 2026-02-22T03:08:57.583 INFO:scrape:suites: ['base', 'cluster/{12-osds}', 'ec_optimizations/ec_optimizations_off', 'mem$/{low}', 'mon_election/connectivity', 'msgr-failures/osd-delay', 'objectstore/{bluestore/{alloc$/{stupid}', 'onode-segment$/{512K}', 'rados', 'rados/thrash-erasure-code-big/{ceph', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashosds-health', 'v1}}}}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'write$/{v1/{compr$/{no$/{no}}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Timeout 3h running clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-22T03:08:57.583 INFO:scrape:1 jobs: ['63178'] 2026-02-22T03:08:57.583 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/misc}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Failure: Command failed (workunit test osd-backfill/osd-backfill-prio.sh) on trial162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh' 2026-02-22T03:08:57.583 INFO:scrape:1 jobs: ['63261'] 2026-02-22T03:08:57.583 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{ubuntu_latest}', 'workloads/osd-backfill}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Assertion: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7fc649385c01] 2026-02-22T03:08:57.583 INFO:scrape:ceph version 20.3.0-5425-g7c8cf961 (7c8cf961761b37084e637a305d42f77c52437802) tentacle (dev - RelWithDebInfo) 1: /lib64/libc.so.6(+0x3fc30) [0x7fc64843fc30] 2: /lib64/libc.so.6(+0x8d03c) [0x7fc64848d03c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18e) [0x7fc649385c01] 6: ceph-bluestore-tool() [0x52a700] 7: (BlueFS::_drop_link_DF(boost::intrusive_ptr)+0xee) [0x6604de] 8: (BlueFS::unlink(std::basic_string_view >, std::basic_string_view >)+0x322) [0x673a72] 9: (BlueRocksEnv::DeleteFile(std::__cxx11::basic_string, std::allocator > const&)+0x47) [0x7b20a7] 10: (rocksdb::DeleteDBFile(rocksdb::ImmutableDBOptions const*, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool)+0x70) [0x94edf0] 11: (rocksdb::DBImpl::DeleteObsoleteFileImpl(int, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, rocksdb::FileType, unsigned long)+0x9c) [0x85563c] 12: (rocksdb::DBImpl::PurgeObsoleteFiles(rocksdb::JobContext&, bool)+0x1666) [0x85a2c6] 13: (rocksdb::DBImpl::DeleteObsoleteFiles()+0x95) [0x85a8c5] 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**, bool, bool)+0x1627) [0x869367] 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string, std::allocator > const&, std::vector > const&, std::vector >*, rocksdb::DB**)+0x28) [0x869988] 16: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string, std::allocator > const&)+0x7ea) [0x6119ca] 17: (BlueStore::_open_db(bool, bool, bool)+0xaf5) [0x6d2b35] 18: (BlueStore::_open_db_and_around(bool, bool)+0x423) [0x6d3333] 19: (BlueStore::expand_devices(std::ostream&)+0xa8d) [0x6dd13d] 20: main() 21: /lib64/libc.so.6(+0x2a610) [0x7fc64842a610] 22: __libc_start_main() 23: _start() 2026-02-22T03:08:57.583 INFO:scrape:1 jobs: ['63287'] 2026-02-22T03:08:57.583 INFO:scrape:suites: ['rados/standalone/{supported-random-distro$/{centos_latest}', 'workloads/osd}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Failure: Test failure: test_sql_autocommit1 (tasks.mgr.test_devicehealth.TestDeviceHealth.test_sql_autocommit1) 2026-02-22T03:08:57.583 INFO:scrape:1 jobs: ['63110'] 2026-02-22T03:08:57.583 INFO:scrape:suites: ['2-ceph', '3-mgrmodules', '4-units/devicehealth}}', 'base', 'debug/mgr', 'distro/{ubuntu_latest}', 'mem$/{normal-1}', 'mgr_ttl_cache/disable', 'mon_election/classic', 'onode-segment$/{none}', 'rados/mgr/{clusters/{2-node-mgr}', 'random-objectstore$/{bluestore/{alloc$/{btree}', 'tasks/{1-install', 'v1}}}}', 'write$/{v1/{compr$/{yes$/{zstd}}'] 2026-02-22T03:08:57.583 INFO:scrape: 2026-02-22T03:08:57.583 INFO:scrape:Failure: Command failed on trial061 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:7c8cf961761b37084e637a305d42f77c52437802-rockylinux-10 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ab2a5c31-0f79-11f1-b4ba-d404e6e7d460 -- bash -c \'set -ex\n# since we don\'"\'"\'t know the real hostnames before the test, the next\n# bit is in order to replace the fake hostnames "host.a/b/c" with\n# the actual names cephadm knows the host by within the mon spec\nceph orch host ls --format json | jq -r \'"\'"\'.[] | .hostname\'"\'"\' > realnames\necho $\'"\'"\'host.a\\nhost.b\\nhost.c\'"\'"\' > fakenames\necho $\'"\'"\'a\\nb\\nc\'"\'"\' > mon_ids\necho $\'"\'"\'{datacenter=a}\\n{datacenter=b,rack=2}\\n{datacenter=a,rack=3}\'"\'"\' > crush_locs\nceph orch ls --service-name mon --export > mon.yaml\nMONSPEC=`cat mon.yaml`\necho "$MONSPEC"\nwhile read realname <&3 && read fakename <&4; do\n MONSPEC="${MONSPEC//$fakename/$realname}"\ndone 3 mon.yaml\ncat mon.yaml\n# now the spec should have the real hostnames, so let\'"\'"\'s re-apply\nceph orch apply -i mon.yaml\nsleep 90\nceph orch ps --refresh\nceph orch ls --service-name mon --export > mon.yaml; ceph orch apply -i mon.yaml\nsleep 90\nceph mon dump\nceph mon dump --format json\n# verify all the crush locations got set from "ceph mon dump" output\nwhile read monid <&3 && read crushloc <&4; do\n ceph mon dump --format json | jq --arg monid "$monid" --arg crushloc "$crushloc" -e \'"\'"\'.mons | .[] | select(.name == $monid) | .crush_location == $crushloc\'"\'"\'\ndone 3 /tmp/mgmt.spec\nservice_type: mgmt-gateway\nservice_id: foo\nplacement:\n hosts:\n - ${HOSTNAME}\nspec:\n ssl_protocols:\n - TLSv1.2\n - TLSv1.3\n ssl_ciphers:\n - AES128-SHA\n - AES256-SHA\n enable_health_check_endpoint: True\nEOT\n# Add generated certificates to spec file\necho " ssl_cert: |" >> /tmp/mgmt.spec\nwhile read LINE; do echo $LINE | sed -e "s/^/ /"; done < /tmp/cert.pem >> /tmp/mgmt.spec\necho " ssl_key: |" >> /tmp/mgmt.spec\nwhile read LINE; do echo $LINE | sed -e "s/^/ /"; done < /tmp/key.pem >> /tmp/mgmt.spec\n# Apply spec\nceph orch apply -i /tmp/mgmt.spec\n\'' 2026-02-22T03:08:57.584 INFO:scrape:1 jobs: ['63182'] 2026-02-22T03:08:57.584 INFO:scrape:suites: ['agent/off', 'mon_election/classic', 'rados/cephadm/workunits/{0-distro/centos_9.stream_runc', 'task/test_mgmt_gateway}'] 2026-02-22T03:08:57.584 INFO:scrape: 2026-02-22T03:08:57.584 INFO:scrape:Failure: Command failed (workunit test dencoder/test-dencoder.sh) on trial046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f8441172c46e34523029ab88f9db7a45514e84 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/dencoder/test-dencoder.sh' 2026-02-22T03:08:57.584 INFO:scrape:1 jobs: ['63066'] 2026-02-22T03:08:57.584 INFO:scrape:suites: ['1-tasks', 'rados/encoder/{0-start', 'supported-random-distro$/{centos_latest}}'] 2026-02-22T03:08:57.584 INFO:scrape: