Found 22 jobs Missing teuthology log /home/teuthworker/archive/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/7658380/teuthology.log Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faee544707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faee544707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faee544707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faa25a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faa25a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faa25a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faa25a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7faa25a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f6b8f84707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f986424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f986424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-3100-g3f7db60e (3f7db60e5ba4c4d053b594b7ec58703d2833c55b) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f986424707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 5 distinct failure reasons Dead: b'2024-04-17T02:41:54.773 INFO:teuthology.orchestra.run.smithi092.stderr:2024-04-17T02:41:54.769+0000 7f94fe0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson' 4 jobs: ['7658389', '7658385', '7658397', '7658393'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/small-objects-localized}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] Dead: None 1 jobs: ['7658380'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2 jobs: ['7658395', '7658387'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}', 'workloads/write_fadvise_dontneed}'] Failure: reached maximum tries (501) after waiting for 3000 seconds 3 jobs: ['7658391', '7658377', '7658379'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2 jobs: ['7658383', '7658381'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects}']