2024-04-16T07:22:25.802 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-16T07:22:25.803 INFO:teuthology.results:seed: '7594'
2024-04-16T07:23:15.225 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-16T07:23:15.229 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-16T19:24:08.825 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2024-04-16T19:24:10.588 INFO:scrape:Found 21 jobs
2024-04-16T19:24:11.567 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f73f2a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.568 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.589 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f73f2a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.590 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.610 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f73f2a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.610 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.629 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f73f2a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.630 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.756 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f39d004707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.756 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.774 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f39d004707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.774 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.939 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f07f8a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.940 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:11.958 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f07f8a4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:11.959 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:12.151 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f015de4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:12.151 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:12.179 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f015de4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:12.179 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:12.938 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fef0444707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:12.938 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:12.963 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fef0444707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:12.963 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.128 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc73f44707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.128 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.155 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc73f44707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.156 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.595 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f4f6d04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.596 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.618 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f4f6d04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.619 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.778 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2139c4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.778 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:13.824 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f2139c4707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:13.824 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:14.288 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f762a04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:14.289 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:14.304 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f762a04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:14.304 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:14.670 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f836f04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:14.671 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:14.687 WARNING:scrape:Ignoring backtrace that exceeds MAX_BACKTRACE_LINES:  ceph version 19.0.0-3109-g30b0aa00 (30b0aa0009a925d6a46536e55c5656edcd51ecdb) squid (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f836f04707a]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598]

2024-04-16T19:24:14.687 WARNING:scrape:Either the backtrace is too long or we did a bad job of checking for end of backtrace!
2024-04-16T19:24:14.699 INFO:scrape:Found 4 distinct failure reasons
2024-04-16T19:24:14.699 INFO:scrape:Failure: reached maximum tries (501) after waiting for 3000 seconds
2024-04-16T19:24:14.700 INFO:scrape:5 jobs: ['7657964', '7657963', '7657960', '7657962', '7657956']
2024-04-16T19:24:14.700 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
2024-04-16T19:24:14.700 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}']
2024-04-16T19:24:14.700 INFO:scrape:
2024-04-16T19:24:14.700 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2024-04-16T19:24:14.700 INFO:scrape:1 jobs: ['7657957']
2024-04-16T19:24:14.700 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}']
2024-04-16T19:24:14.700 INFO:scrape:
2024-04-16T19:24:14.700 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2024-04-16T19:24:14.701 INFO:scrape:10 jobs: ['7657965', '7657976', '7657966', '7657968', '7657969', '7657972', '7657971', '7657967', '7657975', '7657973']
2024-04-16T19:24:14.701 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
2024-04-16T19:24:14.701 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}']
2024-04-16T19:24:14.701 INFO:scrape:
2024-04-16T19:24:14.701 INFO:scrape:Dead: b'2024-04-16T19:23:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2024-04-16T19:23:21.446+0000 7efca479d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson'
2024-04-16T19:24:14.701 INFO:scrape:2 jobs: ['7657974', '7657970']
2024-04-16T19:24:14.701 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
2024-04-16T19:24:14.701 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}', 'workloads/write_fadvise_dontneed}']
2024-04-16T19:24:14.701 INFO:scrape: