2020-11-30T06:57:54.289 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-30T06:57:54.291 INFO:teuthology.results:subset: '1/5' 2020-11-30T06:57:54.291 INFO:teuthology.results:seed: '3716' 2020-11-30T07:10:47.679 INFO:root:teuthology version: 1.0.0-e0aaa9bf 2020-11-30T07:10:47.709 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-11-30T13:40:54.789 INFO:teuthology.results:Tests finished! gathering results... 2020-11-30T13:40:57.436 INFO:scrape:Found 24 jobs 2020-11-30T13:40:58.537 INFO:scrape:Found 3 distinct failure reasons 2020-11-30T13:40:58.538 INFO:scrape:Failure: Command failed on smithi118 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.15-254-g4465473f-1xenial ceph-mds=14.2.15-254-g4465473f-1xenial ceph-mgr=14.2.15-254-g4465473f-1xenial ceph-common=14.2.15-254-g4465473f-1xenial ceph-fuse=14.2.15-254-g4465473f-1xenial ceph-test=14.2.15-254-g4465473f-1xenial radosgw=14.2.15-254-g4465473f-1xenial python-ceph=14.2.15-254-g4465473f-1xenial libcephfs2=14.2.15-254-g4465473f-1xenial libcephfs-dev=14.2.15-254-g4465473f-1xenial librados2=14.2.15-254-g4465473f-1xenial librbd1=14.2.15-254-g4465473f-1xenial rbd-fuse=14.2.15-254-g4465473f-1xenial' 2020-11-30T13:40:58.539 INFO:scrape:5 jobs: ['5668739', '5668722', '5668742', '5668727', '5668723'] 2020-11-30T13:40:58.539 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code-overwrites/{bluestore-bitmap', 'supported-random-distro$/{ubuntu_16.04}', 'thrashosds-health'] 2020-11-30T13:40:58.540 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados:thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'supported-random-distro$/{ubuntu_16.04}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashosds-health', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-overwrites}'] 2020-11-30T13:40:58.540 INFO:scrape: 2020-11-30T13:40:58.541 INFO:scrape:Dead: b'2020-11-30T13:39:12.581 DEBUG:teuthology.exit:Finished running handlers' 2020-11-30T13:40:58.541 INFO:scrape:4 jobs: ['5668737', '5668734', '5668736', '5668741'] 2020-11-30T13:40:58.541 INFO:scrape:suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'rados', 'rados:thrash-erasure-code-overwrites/{bluestore-bitmap', 'supported-random-distro$/{ubuntu_latest}', 'thrashosds-health'] 2020-11-30T13:40:58.542 INFO:scrape:suites union: ['ceph', 'clusters/{fixed-2', 'fast/fast', 'fast/normal', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados:thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/fastread', 'thrashers/pggrow', 'thrashosds-health', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-snaps-few-objects-overwrites}'] 2020-11-30T13:40:58.542 INFO:scrape: 2020-11-30T13:40:58.543 INFO:scrape:Crash: timed out waiting for admin_socket to appear after osd.2 restart 2020-11-30T13:40:58.543 INFO:scrape:ceph version 14.2.15-254-g4465473 (4465473f4beb0abf5eea83640844c5ce6c4a4d1d) nautilus (stable) 1: (()+0x12980) [0x7ff6d4a76980] 2: (gsignal()+0xc7) [0x7ff6d3b54fb7] 3: (abort()+0x141) [0x7ff6d3b56921] 4: (()+0x3048a) [0x7ff6d3b4648a] 5: (()+0x30502) [0x7ff6d3b46502] 6: (crimson::dmclock::PriorityQueueBase, OpQueueItem, false, false, 2u>::do_add_request(std::unique_ptr >&&, std::pair const&, crimson::dmclock::ReqParams const&, double, unsigned int, crimson::dmclock::ClientInfo const&)+0x107c) [0x563005b077ec] 7: (crimson::dmclock::PullPriorityQueue, OpQueueItem, false, false, 2u>::add_request(OpQueueItem&&, std::pair const&, unsigned int)+0x13f) [0x563005b07a2f] 8: (ceph::mClockClientQueue::enqueue(unsigned long, unsigned int, unsigned int, OpQueueItem&&)+0x4c) [0x563005b07b7c] 9: (OSD::ShardedOpWQ::_enqueue(OpQueueItem&&)+0xc3) [0x563005847fd3] 10: (OSD::enqueue_op(spg_t, boost::intrusive_ptr&&, unsigned int)+0x1ca) [0x56300584841a] 11: (OSD::ms_fast_dispatch(Message*)+0x191) [0x56300584d661] 12: (DispatchQueue::fast_dispatch(boost::intrusive_ptr const&)+0x27d) [0x56300624150d] 13: (AsyncConnection::DelayedDelivery::do_request(unsigned long)+0x173) [0x56300626e143] 14: (EventCenter::process_time_events()+0x65a) [0x5630060a38ba] 15: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa00) [0x5630060a8110] 16: (()+0xdc5468) [0x5630060aa468] 17: (()+0xbd6df) [0x7ff6d457a6df] 18: (()+0x76db) [0x7ff6d4a6b6db] 19: (clone()+0x3f) [0x7ff6d3c3771f] 2020-11-30T13:40:58.544 INFO:scrape:1 jobs: ['5668732'] 2020-11-30T13:40:58.544 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'fast/normal', 'msgr-failures/osd-delay', 'openstack}', 'rados', 'rados:thrash-erasure-code-overwrites/{bluestore-bitmap', 'recovery-overrides/{more-active-recovery}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashosds-health', 'workloads/ec-snaps-few-objects-overwrites}'] 2020-11-30T13:40:58.544 INFO:scrape: