Found 358 jobs Ignoring malparsed backtrace: ceph version 16.0.0-5880-g3aa34844 (3aa348442523268d3aa618630ad500f0252d0814) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1b30c488ab] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x55f6a4c99b21] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-24T08:44:33.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-24T08:44:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2020-09-24T08:46:01.810 INFO:tasks.ceph.osd.5:Stopped , 2020-09-24T08:46:01.811 INFO:teuthology.misc:Shutting down mgr daemons... , 2020-09-24T08:46:01.811 DEBUG:tasks.ceph.mgr.x:waiting for process to exit Ignoring malparsed backtrace: ceph version 16.0.0-5880-g3aa34844 (3aa348442523268d3aa618630ad500f0252d0814) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1b30c488ab] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x55f6a4c99b21] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-24T08:44:33.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-24T08:44:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2020-09-24T08:46:01.810 INFO:tasks.ceph.osd.5:Stopped , 2020-09-24T08:46:01.811 INFO:teuthology.misc:Shutting down mgr daemons... , 2020-09-24T08:46:01.811 DEBUG:tasks.ceph.mgr.x:waiting for process to exit Ignoring malparsed backtrace: ceph version 16.0.0-5880-g3aa34844 (3aa348442523268d3aa618630ad500f0252d0814) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1b30c488ab] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x55f6a4c99b21] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-24T08:44:33.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-24T08:44:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2020-09-24T08:46:01.810 INFO:tasks.ceph.osd.5:Stopped , 2020-09-24T08:46:01.811 INFO:teuthology.misc:Shutting down mgr daemons... , 2020-09-24T08:46:01.811 DEBUG:tasks.ceph.mgr.x:waiting for process to exit Ignoring malparsed backtrace: ceph version 16.0.0-5880-g3aa34844 (3aa348442523268d3aa618630ad500f0252d0814) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1b30c488ab] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x55f6a4c99b21] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-24T08:44:33.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-24T08:44:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2020-09-24T08:46:01.810 INFO:tasks.ceph.osd.5:Stopped , 2020-09-24T08:46:01.811 INFO:teuthology.misc:Shutting down mgr daemons... , 2020-09-24T08:46:01.811 DEBUG:tasks.ceph.mgr.x:waiting for process to exit Ignoring malparsed backtrace: ceph version 16.0.0-5880-g3aa34844 (3aa348442523268d3aa618630ad500f0252d0814) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe1) [0x7f1b30c488ab] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x501) [0x55f6a4c99b21] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-24T08:44:33.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-24T08:44:33.549 INFO:teuthology.orchestra.run.smithi016.stdout: Ignoring malparsed backtrace: 2020-09-24T08:46:01.810 INFO:tasks.ceph.osd.5:Stopped , 2020-09-24T08:46:01.811 INFO:teuthology.misc:Shutting down mgr daemons... , 2020-09-24T08:46:01.811 DEBUG:tasks.ceph.mgr.x:waiting for process to exit Found 5 distinct failure reasons Failure: timed out waiting for admin_socket to appear after osd.0 restart 1 jobs: ['5464579'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'rados', 'rados/singleton/{all/rebuild-mondb', 'supported-random-distro$/{rhel_8}}'] Timeout 120 running ceph --cluster ceph --log-early quorum_status 1 jobs: ['5464773'] suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/mon_recovery', 'thrashosds-health}', 'validater/lockdep}'] Failure: "2020-09-24T09:35:35.474625+0000 mds.a (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi053:y (4777), after 301.305 seconds" in cluster log 4 jobs: ['5464755', '5464664', '5464577', '5464776'] suites intersection: [] suites union: ['1-install/nautilus', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{rbd-cls', '5-workload/{radosbench', '6-finish-upgrade', '7-octopus', '8-workload/{rbd-python', 'bluestore-bitmap', 'clusters/{fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/many', 'msgr/async', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/singleton/{all/watch-notify-same-primary', 'rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack', 'rbd-import-export', 'rbd_api}', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'tasks/rados_stress_watch}', 'thrashosds-health', 'ubuntu_latest}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 1 jobs: ['5464654'] suites: ['mon_election/classic', 'msgr-failures/few', 'msgr/async-v1only', 'objectstore/bluestore-hybrid', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{ubuntu_latest}}'] Dead: b'2020-09-24T21:25:37.024 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['5464795'] suites: ['clusters/9-mons', 'mon_election/connectivity', 'msgr-failures/mon-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'rados', 'rados/monthrash/{ceph', 'supported-random-distro$/{rhel_8}', 'thrashers/one', 'workloads/pool-create-delete}']