Found 358 jobs Ignoring malparsed backtrace: ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f79a92a38ab] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x155f) [0x56171065887f] Ignoring malparsed backtrace: 2020-09-23T09:07:17.772 INFO:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-23T09:07:18.061 INFO:teuthology.orchestra.run.smithi093.stdout: , dumped all Ignoring malparsed backtrace: 2020-09-23T09:08:40.184 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-23T09:08:40.210 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-23T09:08:40.211 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2020-09-23T09:10:36.355 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-09-23T09:10:36.367 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-23T09:10:36.368 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f79a92a38ab] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x155f) [0x56171065887f] Ignoring malparsed backtrace: 2020-09-23T09:07:17.772 INFO:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-23T09:07:18.061 INFO:teuthology.orchestra.run.smithi093.stdout: , dumped all Ignoring malparsed backtrace: 2020-09-23T09:08:40.184 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-23T09:08:40.210 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-23T09:08:40.211 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2020-09-23T09:10:36.355 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-09-23T09:10:36.367 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-23T09:10:36.368 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f79a92a38ab] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x155f) [0x56171065887f] Ignoring malparsed backtrace: 2020-09-23T09:07:17.772 INFO:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-23T09:07:18.061 INFO:teuthology.orchestra.run.smithi093.stdout: , dumped all Ignoring malparsed backtrace: 2020-09-23T09:08:40.184 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-23T09:08:40.210 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-23T09:08:40.211 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2020-09-23T09:10:36.355 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-09-23T09:10:36.367 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-23T09:10:36.368 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f79a92a38ab] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x155f) [0x56171065887f] Ignoring malparsed backtrace: 2020-09-23T09:07:17.772 INFO:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-23T09:07:18.061 INFO:teuthology.orchestra.run.smithi093.stdout: , dumped all Ignoring malparsed backtrace: 2020-09-23T09:08:40.184 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-23T09:08:40.210 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-23T09:08:40.211 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2020-09-23T09:10:36.355 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-09-23T09:10:36.367 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-23T09:10:36.368 DEBUG:teuthology.run_tasks:Unwinding manager selinux Ignoring malparsed backtrace: ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f79a92a38ab] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x155f) [0x56171065887f] Ignoring malparsed backtrace: 2020-09-23T09:07:17.772 INFO:teuthology.orchestra.run.smithi093:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-23T09:07:18.061 INFO:teuthology.orchestra.run.smithi093.stdout: , dumped all Ignoring malparsed backtrace: 2020-09-23T09:08:40.184 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-23T09:08:40.210 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-23T09:08:40.211 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: 2020-09-23T09:10:36.355 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab , 2020-09-23T09:10:36.367 INFO:teuthology.task.ansible:Skipping ansible cleanup... , 2020-09-23T09:10:36.368 DEBUG:teuthology.run_tasks:Unwinding manager selinux Found 6 distinct failure reasons Failure: Test failure: test_histogram (tasks.mgr.dashboard.test_osd.OsdTest) 13 jobs: ['5461837', '5461936', '5461799', '5461921', '5461758', '5461997', '5461858', '5461896', '5461819', '5461879', '5461780', '5461976', '5461960'] suites intersection: ['debug/mgr', 'rados/dashboard/{clusters/{2-node-mgr}', 'tasks/dashboard}'] suites union: ['debug/mgr', 'mon_election/classic', 'mon_election/connectivity', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/dashboard}'] Dead: b'2020-09-23T20:15:56.799 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5461700'] suites: ['debug/mgr', 'mon_election/classic', 'objectstore/bluestore-comp-snappy', 'rados/dashboard/{clusters/{2-node-mgr}', 'supported-random-distro$/{centos_8}', 'tasks/dashboard}'] Failure: Command failed (workunit test rados/test.sh) on smithi070 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=277851403934c68710750fb012e2deeafae1bfe3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2 jobs: ['5461935', '5461794'] suites intersection: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] suites union: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/lockdep}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5461755'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-stupid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] Failure: "2020-09-23T08:11:56.838055+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi124:x (4774), after 302.726 seconds" in cluster log 2 jobs: ['5461663', '5461841'] suites intersection: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] suites union: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'mon_election/classic', 'mon_election/connectivity', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'thrashosds-health', 'ubuntu_latest}'] Crash: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_msgr' ceph version 16.0.0-5846-g8fd1bd4a (8fd1bd4a29a81711051eebe2829e967dc4a6f0ff) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7f44d61dc24e] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f44d61dc3d8] 6: (ceph::msgr::v2::FrameAssembler::get_frame_onwire_len() const+0x194) [0x7f44d64d1124] 7: (bool ProtocolV2::append_frame(ceph::msgr::v2::MessageFrame&)+0x84) [0x7f44d64bec84] 8: (ProtocolV2::write_message(Message*, bool)+0x3c9) [0x7f44d64a1a49] 9: (ProtocolV2::write_event()+0x2cb) [0x7f44d64b5bfb] 10: (AsyncConnection::handle_write()+0x43) [0x7f44d6472523] 11: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xdd7) [0x7f44d64c6967] 12: () 13: (std::function::operator()() const+0x32) [0x55cc7ded799e] 14: (void std::__invoke_impl>(std::__invoke_other, std::function&&)+0x20) [0x55cc7e273179] 15: (std::__invoke_result>::type std::__invoke>(std::function&&)+0x35) [0x55cc7e2722f5] 16: (decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker > >::_M_invoke<0ul>(std::_Index_tuple<0ul>)+0x28) [0x55cc7e2741b4] 17: (std::thread::_Invoker > >::operator()()+0x2c) [0x55cc7e274170] 18: (std::thread::_State_impl > > >::_M_run()+0x1c) [0x55cc7e274140] 19: () 20: () 21: clone() 1 jobs: ['5461957'] suites: ['mon_election/classic', 'rados', 'rados/singleton-nomsgr/{all/msgr', 'supported-random-distro$/{ubuntu_latest}}']