Found 96 jobs Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.5.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.4.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-mgr.x.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.2.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.6.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-osd.7.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi008/log/ceph-mgr.y.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.1.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.3.log.gz Missing log /home/teuthworker/archive/teuthology-2020-08-29_03:20:02-krbd-master-testing-basic-smithi/5384879/remote/smithi032/log/ceph-osd.0.log.gz Not-understood crash indication Found 6 distinct failure reasons Failure: "2020-08-30T17:12:16.987267+0000 mon.b (mon.0) 38 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log 2 jobs: ['5384846', '5384916'] suites intersection: ['conf'] suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/object-map', 'krbd/fsx/{ceph/ceph', 'krbd/rbd/{bluestore-bitmap', 'msgr-failures/many', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-3-client}', 'tasks/rbd_workunit_trivial_sync}'] Failure: reached maximum tries (60) after waiting for 900 seconds 2 jobs: ['5384842', '5384909'] Crash: Command failed (workunit test rbd/huge-tickets.sh) on smithi188 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ed90c0768b0c8cf09511e8971bba453e09e42697 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/huge-tickets.sh' ceph version 16.0.0-4846-ged90c0768b (ed90c0768b0c8cf09511e8971bba453e09e42697) pacific (dev) 1: (()+0x128a0) [0x7f7c2b52b8a0] 2: (gsignal()+0xc7) [0x7f7c2a623f47] 3: (abort()+0x141) [0x7f7c2a6258b1] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f7c2c6b1e1a] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f7c2c6b1fa4] 6: (ProtocolV2::write(std::__cxx11::basic_string, std::allocator > const&, CtFun&, ceph::buffer::v15_2_0::list&)+0x1ac) [0x7f7c2c97a37c] 7: (Ct* ProtocolV2::write(std::__cxx11::basic_string, std::allocator > const&, CtFun&, ceph::msgr::v2::AuthDoneFrame&)+0x18b) [0x7f7c2c9967db] 8: (ProtocolV2::_handle_auth_request(ceph::buffer::v15_2_0::list&, bool)+0x34a) [0x7f7c2c986a5a] 9: (ProtocolV2::handle_auth_request_more(ceph::buffer::v15_2_0::list&)+0x23e) [0x7f7c2c9877fe] 10: (ProtocolV2::handle_frame_payload()+0xfb) [0x7f7c2c98fc8b] 11: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x7f7c2c98fff0] 12: (ProtocolV2::_handle_read_frame_epilogue_main()+0x69) [0x7f7c2c990159] 13: (ProtocolV2::_handle_read_frame_segment()+0x121) [0x7f7c2c990511] 14: (ProtocolV2::handle_read_frame_segment(std::unique_ptr&&, int)+0xd7) [0x7f7c2c9916f7] 15: (ProtocolV2::run_continuation(Ct&)+0x34) [0x7f7c2c979c64] 16: (AsyncConnection::process()+0x5fc) [0x7f7c2c944dfc] 17: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0x7dd) [0x7f7c2c99ab5d] 18: (()+0x554920) [0x7f7c2c9a2920] 19: (std::function::operator()() const+0x32) [0x55cbea1b7e1c] 20: (void std::__invoke_impl>(std::__invoke_other, std::function&&)+0x20) [0x55cbea551511] 21: (std::__invoke_result>::type std::__invoke>(std::function&&)+0x35) [0x55cbea55068d] 22: (decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker > >::_M_invoke<0ul>(std::_Index_tuple<0ul>)+0x28) [0x55cbea55254c] 23: (std::thread::_Invoker > >::operator()()+0x2c) [0x55cbea552508] 24: (std::thread::_State_impl > > >::_M_run()+0x1c) [0x55cbea5524d8] 25: (()+0xbd6df) [0x7f7c2b0496df] 26: (()+0x76db) [0x7f7c2b5206db] 27: (clone()+0x3f) [0x7f7c2a706a3f] 2 jobs: ['5384904', '5384855'] suites intersection: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'tasks/rbd_huge_tickets}'] suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_huge_tickets}'] Failure: Command failed on smithi098 with status 3: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeh8kcfytg -r -- '-g auto -g blockdev -x clone'" 2 jobs: ['5384876', '5384827'] suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] Dead: b'2020-08-31T05:11:47.593 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5384879'] suites: ['clusters/3-node', 'conf', 'features/no-object-map', 'krbd/fsx/{ceph/ceph', 'objectstore/filestore-xfs', 'randomized-striping-on}', 'striping/fancy/{msgr-failures/few', 'tasks/fsx-3-client}'] Failure: "2020-08-30T17:48:46.655835+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4444.0:8611 2.2 2:52b97c0e:::rbd_data.11532a8deda0.00000000000002cf:head [set-alloc-hint object_size 4194304 write_size 4194304,write 983040~28672 in=28672b] snapc 0=[] ondisk+write+known_if_redirected e23) initiated 2020-08-30T17:48:16.346885+0000 currently waiting for sub ops" in cluster log 1 jobs: ['5384906'] suites: ['clusters/fixed-3', 'conf', 'krbd/rbd/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_workunit_suites_fsx}']