2022-04-25T14:16:11.357 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-25T14:16:11.357 INFO:teuthology.results:subset: '111/120000' 2022-04-25T14:16:11.357 INFO:teuthology.results:seed: '8236' 2022-04-25T14:28:29.973 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-25T14:28:29.977 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-25T20:56:56.812 INFO:teuthology.results:Tests finished! gathering results... 2022-04-25T20:56:58.409 INFO:scrape:Found 20 jobs 2022-04-25T20:56:58.492 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.0-58-ge2161634 (e21616343a03209745d1ef7ffb6304159addf4fd) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f877495e76f] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x55a6be0f34e4] 2022-04-25T20:56:58.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:34:47.128 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-04-25T14:34:47.129 INFO:teuthology.orchestra.run.smithi082.stdout:} , 2022-04-25T14:34:47.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-04-25T20:56:58.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:35:50.106 INFO:teuthology.orchestra.run:waiting for 300 , 2022-04-25T14:35:50.163 INFO:tasks.ceph.osd.4:Stopped , 2022-04-25T14:35:50.164 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-04-25T20:56:58.494 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 2 6 377 2 -1227us[-1183us] +/- 50ms , 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +5316us[+5359us] +/- 81ms , 2022-04-25T14:36:57.306 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 3 +3816us[+3860us] +/- 86ms 2022-04-25T20:56:58.579 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.0-58-ge2161634 (e21616343a03209745d1ef7ffb6304159addf4fd) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f877495e76f] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x55a6be0f34e4] 2022-04-25T20:56:58.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:34:47.128 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-04-25T14:34:47.129 INFO:teuthology.orchestra.run.smithi082.stdout:} , 2022-04-25T14:34:47.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-04-25T20:56:58.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:35:50.106 INFO:teuthology.orchestra.run:waiting for 300 , 2022-04-25T14:35:50.163 INFO:tasks.ceph.osd.4:Stopped , 2022-04-25T14:35:50.164 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-04-25T20:56:58.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 2 6 377 2 -1227us[-1183us] +/- 50ms , 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +5316us[+5359us] +/- 81ms , 2022-04-25T14:36:57.306 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 3 +3816us[+3860us] +/- 86ms 2022-04-25T20:56:58.672 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.2.0-58-ge2161634 (e21616343a03209745d1ef7ffb6304159addf4fd) quincy (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f877495e76f] , 2: (CopyFromOp::_finish(TestOp::CallbackInfo*)+0x4f4) [0x55a6be0f34e4] 2022-04-25T20:56:58.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:34:47.128 INFO:teuthology.orchestra.run.smithi082.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-04-25T14:34:47.129 INFO:teuthology.orchestra.run.smithi082.stdout:} , 2022-04-25T14:34:47.142 DEBUG:teuthology.orchestra.run.smithi082:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-04-25T20:56:58.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:35:50.106 INFO:teuthology.orchestra.run:waiting for 300 , 2022-04-25T14:35:50.163 INFO:tasks.ceph.osd.4:Stopped , 2022-04-25T14:35:50.164 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-04-25T20:56:58.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv02.front.sepia.ceph.com 2 6 377 2 -1227us[-1183us] +/- 50ms , 2022-04-25T14:36:57.305 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +5316us[+5359us] +/- 81ms , 2022-04-25T14:36:57.306 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 3 +3816us[+3860us] +/- 86ms 2022-04-25T20:56:58.688 INFO:scrape:Found 2 distinct failure reasons 2022-04-25T20:56:58.688 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2022-04-25T20:56:58.688 INFO:scrape:1 jobs: ['6805194'] 2022-04-25T20:56:58.688 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] 2022-04-25T20:56:58.689 INFO:scrape: 2022-04-25T20:56:58.689 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 10000 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base' 2022-04-25T20:56:58.689 INFO:scrape:1 jobs: ['6805186'] 2022-04-25T20:56:58.689 INFO:scrape:suites: ['mon_election/connectivity', 'msgr-failures/few', 'msgr/async', 'objectstore/bluestore-comp-zlib', 'rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'supported-random-distro$/{centos_8}}'] 2022-04-25T20:56:58.689 INFO:scrape: