Found 200 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f49fc8de520] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7f49fd0d1b24] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f49fc8de520] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7f49fd0d1b24] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f49fc8de520] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7f49fd0d1b24] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-432-g3cfd0d29 (3cfd0d29cc3905ea860fc5482a5a58407e15a0d7) squid (dev) , 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f49fc8de520] , 2: (ceph::common::PerfCounters::inc(int, unsigned long)+0x4) [0x7f49fd0d1b24] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Missing teuthology log /home/teuthworker/archive/pdiazbou-2024-02-02_19:09:44-rados-wip-pereman2-buffer-space-distro-default-smithi/7544682/teuthology.log Found 10 distinct failure reasons Dead: b'2024-02-04T01:40:52.807 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['7544611', '7544621', '7544721'] suites intersection: ['rados'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}}', 'thrashers/careful', 'thrashers/none', 'thrashosds-health', 'workloads/cache-agent-big}', 'workloads/cache-snaps}'] Failure: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2 jobs: ['7544697', '7544620'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'validater/valgrind}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/none', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'tasks/rados_cls_all', 'validater/valgrind}'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/0 --gtest_catch_exceptions=0\'' 1 jobs: ['7544740'] suites: ['rados/objectstore/{backends/objectstore-memstore', 'supported-random-distro$/{ubuntu_latest}}'] Failure: reached maximum tries (51) after waiting for 300 seconds 1 jobs: ['7544675'] suites: ['2-inject-leak/osd', 'centos_latest}', 'rados/valgrind-leaks/{1-start'] Failure: Command failed on smithi176 with status 1: 'test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm' 1 jobs: ['7544613'] suites: ['0-start', '1-tasks', 'mon_election/connectivity', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools_crun}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'test_rbd_api', 'test_rbd_python}}', 'upgrade-sequence', 'workload/{ec-rados-default'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' 1 jobs: ['7544716'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'supported-random-distro$/{ubuntu_latest}}'] Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 1 jobs: ['7544582'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/classic', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_cls_all', 'thrashosds-health}', 'validater/lockdep}'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi079 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 1 jobs: ['7544702'] suites: ['debug/mgr', 'mon_election/classic', 'rados/dashboard/{0-single-container-host', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/e2e}'] Failure: Command failed (workunit test cephtool/test.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' 1 jobs: ['7544661'] suites: ['mon_election/classic', 'msgr-failures/none', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'rados', 'rados/singleton-bluestore/{all/cephtool', 'supported-random-distro$/{ubuntu_latest}}'] Dead: None 1 jobs: ['7544682']