2020-09-08T15:56:20.974 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-08T15:56:20.976 INFO:teuthology.results:seed: '2477' 2020-09-08T15:56:36.695 INFO:root:teuthology version: 1.0.0-55bd2dc 2020-09-08T15:56:36.697 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2020-09-08T16:08:37.008 INFO:teuthology.results:Tests finished! gathering results... 2020-09-08T16:08:37.077 INFO:scrape:Found 1 jobs 2020-09-08T16:08:37.103 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5243-gb73f287 (b73f2876499ccc7ac6fb37b6aebfcf4e3293d037) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f356e02d418] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55b899cc5e62] 2020-09-08T16:08:37.107 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:06.704 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub , reweighted osd.1 to 1 (10000) , nodeep-scrub is unset 2020-09-08T16:08:37.109 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:25.135 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout:{ , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "" 2020-09-08T16:08:37.110 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:47.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi106.front.sepia.ceph.com , 2020-09-08T16:07:47.371 INFO:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2020-09-08T16:07:47.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi013.front.sepia.ceph.com 2020-09-08T16:08:37.111 WARNING:scrape:Ignoring malparsed backtrace: Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service , 2020-09-08T16:07:49.472 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2020-09-08T16:08:37.153 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5243-gb73f287 (b73f2876499ccc7ac6fb37b6aebfcf4e3293d037) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f356e02d418] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55b899cc5e62] 2020-09-08T16:08:37.156 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:06.704 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub , reweighted osd.1 to 1 (10000) , nodeep-scrub is unset 2020-09-08T16:08:37.158 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:25.135 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout:{ , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "" 2020-09-08T16:08:37.159 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:47.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi106.front.sepia.ceph.com , 2020-09-08T16:07:47.371 INFO:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2020-09-08T16:07:47.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi013.front.sepia.ceph.com 2020-09-08T16:08:37.160 WARNING:scrape:Ignoring malparsed backtrace: Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service , 2020-09-08T16:07:49.472 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2020-09-08T16:08:37.193 WARNING:scrape:Ignoring malparsed backtrace: ceph version 16.0.0-5243-gb73f287 (b73f2876499ccc7ac6fb37b6aebfcf4e3293d037) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f356e02d418] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55b899cc5e62] 2020-09-08T16:08:37.196 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:06.704 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd unset nodeep-scrub , reweighted osd.1 to 1 (10000) , nodeep-scrub is unset 2020-09-08T16:08:37.197 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:25.135 INFO:teuthology.orchestra.run.smithi106:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout:{ , 2020-09-08T16:07:25.254 INFO:teuthology.orchestra.run.smithi106.stdout: "success": "" 2020-09-08T16:08:37.198 WARNING:scrape:Ignoring malparsed backtrace: 2020-09-08T16:07:47.370 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi106.front.sepia.ceph.com , 2020-09-08T16:07:47.371 INFO:teuthology.orchestra.run.smithi106:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 , 2020-09-08T16:07:47.431 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi013.front.sepia.ceph.com 2020-09-08T16:08:37.198 WARNING:scrape:Ignoring malparsed backtrace: Redirecting to /bin/systemctl restart rsyslog.service , Redirecting to /bin/systemctl restart rsyslog.service , 2020-09-08T16:07:49.472 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2020-09-08T16:08:37.222 INFO:scrape:Found 1 distinct failure reasons 2020-09-08T16:08:37.224 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2020-09-08T16:08:37.224 INFO:scrape:1 jobs: ['5418060'] 2020-09-08T16:08:37.225 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2020-09-08T16:08:37.226 INFO:scrape: 2020-09-08T16:08:37.226 INFO:teuthology.results:Sending results to tchaikov@gmail.com: Test Run: kchai-2020-09-08_15:56:07-crimson-rados-wip-37045-37042-kefu-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/kchai-2020-09-08_15:56:07-crimson-rados-wip-37045-37042-kefu-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-08_15:56:07-crimson-rados-wip-37045-37042-kefu-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5418060] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:06:13 info: http://pulpito.ceph.com/kchai-2020-09-08_15:56:07-crimson-rados-wip-37045-37042-kefu-distro-basic-smithi/5418060/ log: http://qa-proxy.ceph.com/teuthology/kchai-2020-09-08_15:56:07-crimson-rados-wip-37045-37042-kefu-distro-basic-smithi/5418060/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'