Found 277 jobs Missing teuthology log /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51512/teuthology.log Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc7) [0x7f90201225d0] , 2: ceph_test_rados(+0x2e9bb) [0x55d898e189bb] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 11 distinct failure reasons Crash: Command failed (workunit test cephtool/test_kvstore_tool.sh) on trial073 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh' ceph version 19.2.3-675-g3c96dfb3 (3c96dfb3f50b3d02598a82af7104764bfdcd3919) squid (stable) 1: /lib64/libc.so.6(+0x3fc30) [0x7fbdcf03fc30] 2: (RocksDBStore::get_wholespace_iterator(unsigned int)+0xb4) [0x5634a5ac12d4] 3: (RocksDBStore::get_iterator(std::__cxx11::basic_string, std::allocator > const&, unsigned int, KeyValueDB::IteratorBounds)+0x240) [0x5634a5ac19f0] 4: (BlueStore::_close_db()+0x380) [0x5634a58fe8a0] 5: (BlueStore::umount()+0x244) [0x5634a590b414] 6: ceph-kvstore-tool(+0x1f7553) [0x5634a58a8553] 7: main() 8: /lib64/libc.so.6(+0x2a610) [0x7fbdcf02a610] 9: __libc_start_main() 10: _start() 2 jobs: ['51481', '51346'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-kvstore-tool', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on trial138 with status 1: 'sudo yum -y install ceph-test' 4 jobs: ['51483', '51487', '51396', '51480'] suites intersection: [] suites union: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/quincy', '1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'clusters/{fixed-4', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-delay', 'msgr/async-v1only', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/dashboard/{0-single-container-host', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'random-objectstore$/{bluestore-comp-lz4}', 'recovery-overrides/{more-async-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'tasks/e2e}', 'thrashers/default', 'thrashers/none', 'thrashers/pggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/small-objects-localized}', 'workloads/test_rbd_api}'] Failure: ['Error getting key from: https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub'] 12 jobs: ['51499', '51502', '51496', '51491', '51494', '51493', '51501', '51500', '51498', '51492', '51497', '51495'] suites intersection: [] suites union: ['1-pg-log-overrides/short_pg_log', '1-start', '2-ceph', '2-recovery-overrides/{more-active-recovery}', '2-services/rgw', '3-final}', '3-mgrmodules', '3-scrub-overrides/{max-simultaneous-scrubs-1}', '4-units/prometheus}}', 'backoff/peering', 'ceph', 'cluster/{12-osds', 'clusters/fixed-2', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/read', 'd-thrash/default/{default', 'debug/mgr', 'distro/{centos_latest}', 'fast/fast', 'mgr_ttl_cache/disable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc}', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/multimon/{clusters/21', 'rados/singleton-nomsgr/{all/ceph-snapmapper', 'rados/singleton/{all/thrash-backfill-full', 'rados/singleton/{all/thrash-eio', 'rados/standalone/{supported-random-distro$/{centos_latest}', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-comp-snappy}', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_clock_no_skews}', 'tasks/mon_recovery', 'tasks/{1-install', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashosds-health', 'thrashosds-health}', 'validater/lockdep}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-small-objects-overwrites}', 'workloads/mon}', 'workloads/small-objects}'] Failure: Command failed on trial074 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3c96dfb3f50b3d02598a82af7104764bfdcd3919' 3 jobs: ['51485', '51469', '51330'] suites intersection: [] suites union: ['0-start', '1-tasks', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/none', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'overrides/ignorelist_health', 'rados', 'rados/singleton/{all/test-noautoscale-flag', 'rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'supported-random-distro$/{centos_latest}}', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Dead: None 50 jobs Failure: Command failed (workunit test cephtool/test.sh) on trial112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 3 jobs: ['51453', '51388', '51319'] suites intersection: ['rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{centos_latest}}'] Failure: "2026-02-17T11:55:42.442868+0000 mon.a (mon.0) 338 : cluster [WRN] Health check failed: 1 OSD(s) experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT)" in cluster log 8 jobs: ['51344', '51386', '51309', '51461', '51331', '51463', '51482', '51403'] suites intersection: [] suites union: ['0-nvme-loop', '2-inject-leak/mon', 'agent/off', 'centos_latest}', 'ceph', 'clusters/fixed-2', 'clusters/fixed-4', 'd-thrash/default/{default', 'd-thrash/none', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/none', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'rados', 'rados/cephadm/smoke/{0-distro/ubuntu_22.04', 'rados/multimon/{clusters/9', 'rados/singleton/{all/mon-stretch-pool', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'start}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'tasks/mon_recovery', 'tasks/mon_recovery}', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/lockdep}', 'validater/valgrind}'] Failure: "2026-02-17T13:51:14.547427+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 126427.370278 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.0. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 6 jobs: ['51561', '51408', '51443', '51447', '51365', '51323'] suites intersection: ['rados'] suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/quincy', '1-install/reef', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'rados', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/thrash-old-clients/{0-distro$/{centos_9.stream}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/default', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/radosbench}', 'workloads/rbd_cls}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 1 jobs: ['51405'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/upmap-read', 'mon_election/classic', 'msgr-failures/fastclose', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/dedup-io-snaps}'] Failure: Command failed (workunit test post-file.sh) on trial054 with status 255: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c96dfb3f50b3d02598a82af7104764bfdcd3919 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/post-file.sh' 2 jobs: ['51490', '51354'] suites intersection: ['rados', 'rados/singleton-nomsgr/{all/ceph-post-file'] suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados', 'rados/singleton-nomsgr/{all/ceph-post-file', 'supported-random-distro$/{centos_latest}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on trial192 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 1 jobs: ['51488'] suites: ['mon_election/classic', 'objectstore/bluestore-comp', 'openstack', 'rados/perf/{ceph', 'scheduler/dmclock_default_shards', 'settings/optimized', 'ubuntu_latest', 'workloads/fio_4K_rand_read}']