Found 34 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f89d7a319fc] , 2: ceph_test_rados(+0x1b83a) [0x5641b10d983a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f89d7a319fc] , 2: ceph_test_rados(+0x1b83a) [0x5641b10d983a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f89d7a319fc] , 2: ceph_test_rados(+0x1b83a) [0x5641b10d983a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7f89d7a319fc] , 2: ceph_test_rados(+0x1b83a) [0x5641b10d983a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7ffbcea319fc] , 2: ceph_test_rados(+0x1b83a) [0x5567edd2783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-635-g4e3957a1 (4e3957a13c3b8938fe41053b13fb80161e1916b7) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7ffbcea319fc] , 2: ceph_test_rados(+0x1b83a) [0x5567edd2783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 10 distinct failure reasons Dead: b'2024-01-11T22:01:32.681 INFO:teuthology.orchestra.run.smithi017.stderr:2024-01-11T22:01:32.679+0000 7f3275e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 5 jobs: ['7513441', '7513450', '7513445', '7513442', '7513447'] suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/basic/{clusters/fixed-2', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'tasks/readwrite}', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}', 'workloads/write_fadvise_dontneed}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['7513418'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] Dead: b'2024-01-11T22:01:02.929 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['7513433', '7513424', '7513436'] suites intersection: ['crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/perf/{clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench_4K_seq_read}', 'workloads/small-objects-balanced}'] Failure: reached maximum tries (501) after waiting for 3000 seconds 1 jobs: ['7513429'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}'] Timeout 120 running ceph --cluster ceph status --format=json 1 jobs: ['7513443'] suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests}'] Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2 jobs: ['7513426', '7513420'] suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \ 1 jobs: ['7513448'] suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2 jobs: ['7513435', '7513439'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}', 'workloads/small-objects}'] Timeout 120 running ceph --cluster ceph pg dump --format=json 1 jobs: ['7513432'] suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] Timeout 1h running clone.client.0/qa/workunits/rados/test_python.sh -m \ 1 jobs: ['7513431'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}']