Found 34 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-749-gf111080c (f111080cc966f2a0d01f3e9e3b0f671b4e55cf2f) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7194316e8] , 2: ceph_test_rados(+0x1b83a) [0x565324de783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-749-gf111080c (f111080cc966f2a0d01f3e9e3b0f671b4e55cf2f) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7194316e8] , 2: ceph_test_rados(+0x1b83a) [0x565324de783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-749-gf111080c (f111080cc966f2a0d01f3e9e3b0f671b4e55cf2f) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7194316e8] , 2: ceph_test_rados(+0x1b83a) [0x565324de783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-749-gf111080c (f111080cc966f2a0d01f3e9e3b0f671b4e55cf2f) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcf) [0x7fd7194316e8] , 2: ceph_test_rados(+0x1b83a) [0x565324de783a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 9 distinct failure reasons Dead: b'2024-01-22T16:25:48.275 INFO:tasks.ceph.mgr.x.smithi179.stderr:2024-01-22T16:25:48.293+0000 7f1b885f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29576) UID: 0' 1 jobs: ['7526242'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/readwrite}'] Timeout 1h running clone.client.0/qa/workunits/rados/test_python.sh -m \ 1 jobs: ['7526231'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_python}'] Failure: Command failed (workunit test rados/test.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc936b4ffe3c76c1a6912a55c28e862471635c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 1 jobs: ['7526218'] suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] Dead: b'2024-01-23T04:06:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:2024-01-23T04:06:04.116+0000 7fc862aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 2 jobs: ['7526245', '7526247'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] Timeout 120 running ceph --cluster ceph osd dump --format=json 1 jobs: ['7526235'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] Dead: b'2024-01-23T03:57:55.227 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7526224'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}'] Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2 jobs: ['7526226', '7526220'] suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \ 1 jobs: ['7526248'] suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --write-fadvise-dontneed --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['7526250'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/write_fadvise_dontneed}']