Found 34 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f3d79c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f3d79c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f3d79c4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa88924707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa88924707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa88924707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fa88924707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7ffa2d04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7ffa2d04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Missing teuthology log /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629051/teuthology.log Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f57c344707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fd504e4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2473-g785fc5e8 (785fc5e8d093374f751c7da3e51e9b56b3718d17) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fd504e4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1bc) [0x44f17a] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 4 distinct failure reasons Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2 jobs: ['7629061', '7629055'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 3 jobs: ['7629046', '7629063', '7629069'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}'] Dead: None 2 jobs: ['7629051', '7629073'] Failure: Command failed on smithi137 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2 jobs: ['7629066', '7629071'] suites intersection: ['crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized'] suites union: ['crimson-rados/perf/{clusters/{fixed-2}', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'settings/optimized', 'workloads/radosbench_4M_write}', 'workloads/sample_fio}']