2023-03-23T20:21:15.445 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-23T20:21:15.446 INFO:teuthology.results:subset: '111/120000' 2023-03-23T20:21:15.446 INFO:teuthology.results:seed: '5912' 2023-03-23T20:26:41.392 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-23T20:26:41.397 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-23T21:07:44.741 INFO:teuthology.results:Tests finished! gathering results... 2023-03-23T21:07:45.038 INFO:scrape:Found 5 jobs 2023-03-23T21:07:45.335 INFO:scrape:Found 2 distinct failure reasons 2023-03-23T21:07:45.335 INFO:scrape:Crash: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' 2023-03-23T21:07:45.336 INFO:scrape:ceph version 18.0.0-3070-gfe12066b (fe12066bd59aabf26668e7dd51d06f3a1db19128) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f362f7ac420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x562ca011c1c1] 5: ceph-osd(+0xeef323) [0x562ca011c323] 6: (BlueStore::Collection::make_blob_shared(unsigned long, boost::intrusive_ptr)+0x42e) [0x562ca06e893e] 7: (BlueStore::ExtentMap::make_range_shared(BlueStore*, BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long)+0x206) [0x562ca0716916] 8: (BlueStore::ExtentMap::dup(BlueStore*, BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long&, unsigned long&, unsigned long&)+0x1f1) [0x562ca073fca1] 9: (BlueStore::_do_clone_range(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, boost::intrusive_ptr&, unsigned long, unsigned long, unsigned long)+0x10e) [0x562ca074140e] 10: (BlueStore::_clone(BlueStore::TransContext*, boost::intrusive_ptr&, boost::intrusive_ptr&, boost::intrusive_ptr&)+0x8cb) [0x562ca0765f2b] 11: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x1054) [0x562ca07770a4] 12: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2df) [0x562ca077850f] 13: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x562ca03a6ba3] 14: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr, ECSubWrite&, ZTracer::Trace const&)+0x1064) [0x562ca06328b4] 15: (ECBackend::try_reads_to_commit()+0x2ad3) [0x562ca0642023] 16: (ECBackend::check_ops()+0x24) [0x562ca0642ab4] 17: (CallClientContexts::finish(std::pair&)+0x996) [0x562ca065f356] 18: (ECBackend::complete_read_op(ECBackend::ReadOp&, RecoveryMessages*)+0x8d) [0x562ca062d4cd] 19: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xdc2) [0x562ca0644aa2] 20: (ECBackend::_handle_message(boost::intrusive_ptr)+0x2d4) [0x562ca0645304] 21: (PGBackend::handle_message(boost::intrusive_ptr)+0x56) [0x562ca03f03f6] 22: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x4d2) [0x562ca038beb2] 23: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562ca01de946] 24: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562ca04e72d9] 25: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xb38) [0x562ca01f6168] 26: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x43b) [0x562ca089116b] 27: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562ca0894274] 28: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f362f7a0609] 29: clone() 2023-03-23T21:07:45.336 INFO:scrape:1 jobs: ['7218221'] 2023-03-23T21:07:45.337 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}'] 2023-03-23T21:07:45.337 INFO:scrape: 2023-03-23T21:07:45.337 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 50 --op append_excl 50 --pool unique_pool_0' 2023-03-23T21:07:45.337 INFO:scrape:1 jobs: ['7218219'] 2023-03-23T21:07:45.338 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'openstack}', 'rados', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-small-objects-overwrites}'] 2023-03-23T21:07:45.338 INFO:scrape: