Found 21 jobs Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f413d04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f413d04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f413d04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f050884707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fde8024707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x129) [0x7ff7fd046968] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff7fd046bae] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fb1bf44707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f94e9a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f20eaa4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f70bea4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f5ad5a4707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fc1a804707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f018e04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Ignoring backtrace that exceeds MAX_BACKTRACE_LINES: ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fb5cc04707a] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x5da) [0x44f598] Either the backtrace is too long or we did a bad job of checking for end of backtrace! Found 7 distinct failure reasons Timeout 120 running ceph --cluster ceph osd dump --format=json 1 jobs: ['7652216'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 12 jobs: ['7652229', '7652232', '7652234', '7652227', '7652230', '7652224', '7652225', '7652228', '7652233', '7652223', '7652226', '7652231'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] Failure: reached maximum tries (501) after waiting for 3000 seconds 3 jobs: ['7652221', '7652220', '7652215'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}', 'workloads/radosbench}'] Failure: Command failed on smithi142 with status 1: 'sudo yum -y install cephfs-java' 2 jobs: ['7652219', '7652218'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Timeout 120 running ceph --cluster ceph osd last-stat-seq osd.2 1 jobs: ['7652217'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench-high-concurrency}'] Timeout 120 running ceph --cluster ceph osd last-stat-seq osd.0 1 jobs: ['7652214'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] Crash: reached maximum tries (501) after waiting for 3000 seconds ceph version 19.0.0-2467-g885c5fc4 (885c5fc4bc04c308beb5d00c1cd2cbccedc70e1e) squid (dev) 1: ceph-mgr() [0xa0fcaf] 2: /lib64/libc.so.6(+0x54db0) [0x7f8e23054db0] 3: /lib64/libc.so.6(+0xa154c) [0x7f8e230a154c] 4: raise() 5: abort() 6: /lib64/libstdc++.so.6(+0xa1a01) [0x7f8e254a1a01] 7: /lib64/libstdc++.so.6(+0xad37c) [0x7f8e254ad37c] 8: /lib64/libstdc++.so.6(+0xad3e7) [0x7f8e254ad3e7] 9: /lib64/libstdc++.so.6(+0xad649) [0x7f8e254ad649] 10: (ceph::buffer::v15_2_0::list::iterator_impl::copy(unsigned int, ceph::buffer::v15_2_0::list&)+0x5a) [0x7f8e244a90d8] 11: (ceph::decode(ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list::iterator_impl&)+0x68) [0x7f8e241b9a4f] 12: (OSDMap::Incremental::decode(ceph::buffer::v15_2_0::list::iterator_impl&)+0x559) [0x7f8e24566b7f] 13: /lib64/librados.so.2(+0x1136e3) [0x7f8e2332e6e3] 14: /lib64/librados.so.2(+0xe3a95) [0x7f8e232fea95] 15: /lib64/librados.so.2(+0xe7855) [0x7f8e23302855] 16: /lib64/librados.so.2(+0xaa8a2) [0x7f8e232c58a2] 17: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0xa8) [0x7f8e243866be] 18: (DispatchQueue::entry()+0x44b) [0x7f8e243827db] 19: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e24437c85] 20: (Thread::entry_wrapper()+0x43) [0x7f8e24224b65] 21: (Thread::_entry_func(void*)+0xd) [0x7f8e24224b81] 22: /lib64/libc.so.6(+0x9f802) [0x7f8e2309f802] 23: /lib64/libc.so.6(+0x3f450) [0x7f8e2303f450] 1 jobs: ['7652222'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/radosbench}']