2021-03-07T14:46:30.147 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-07T14:46:30.150 INFO:teuthology.results:seed: '2477' 2021-03-07T14:47:43.002 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-07T14:47:43.010 INFO:teuthology.results:Waiting up to 7200 seconds for tests to finish... 2021-03-07T15:05:44.370 INFO:teuthology.results:Tests finished! gathering results... 2021-03-07T15:05:44.477 INFO:scrape:Found 1 jobs 2021-03-07T15:05:44.710 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1644-gd9ef2504 (d9ef2504f1ddfcbe9c7db5ca1a7e53faf4a254be) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f962c051638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55d98e71f552] 2021-03-07T15:05:44.712 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T15:04:43.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2021-03-07T15:04:44.213 INFO:teuthology.orchestra.run.smithi181.stdout:34359738397 , 2021-03-07T15:04:44.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738397 got 34359738397 for osd.3 2021-03-07T15:05:44.713 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2021-03-07T15:05:44.713 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' , 2021-03-07T15:05:11.809 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... , 2021-03-07T15:05:11.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-03-07T15:05:44.715 WARNING:scrape:Ignoring malparsed backtrace: fetch_binaries_for_coredumps(path, rem) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps , dump_program = dump_out.split("from '")[1].split(' ')[0] 2021-03-07T15:05:44.735 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1644-gd9ef2504 (d9ef2504f1ddfcbe9c7db5ca1a7e53faf4a254be) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f962c051638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55d98e71f552] 2021-03-07T15:05:44.737 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T15:04:43.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2021-03-07T15:04:44.213 INFO:teuthology.orchestra.run.smithi181.stdout:34359738397 , 2021-03-07T15:04:44.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738397 got 34359738397 for osd.3 2021-03-07T15:05:44.738 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2021-03-07T15:05:44.739 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' , 2021-03-07T15:05:11.809 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... , 2021-03-07T15:05:11.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-03-07T15:05:44.739 WARNING:scrape:Ignoring malparsed backtrace: fetch_binaries_for_coredumps(path, rem) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps , dump_program = dump_out.split("from '")[1].split(' ')[0] 2021-03-07T15:05:44.760 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1644-gd9ef2504 (d9ef2504f1ddfcbe9c7db5ca1a7e53faf4a254be) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f962c051638] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x6e2) [0x55d98e71f552] 2021-03-07T15:05:44.763 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-07T15:04:43.906 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2021-03-07T15:04:44.213 INFO:teuthology.orchestra.run.smithi181.stdout:34359738397 , 2021-03-07T15:04:44.227 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738397 got 34359738397 for osd.3 2021-03-07T15:05:44.763 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2021-03-07T15:05:44.765 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' , 2021-03-07T15:05:11.809 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... , 2021-03-07T15:05:11.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-03-07T15:05:44.766 WARNING:scrape:Ignoring malparsed backtrace: fetch_binaries_for_coredumps(path, rem) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps , dump_program = dump_out.split("from '")[1].split(' ')[0] 2021-03-07T15:05:44.772 INFO:scrape:Found 1 distinct failure reasons 2021-03-07T15:05:44.773 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 2021-03-07T15:05:44.773 INFO:scrape:1 jobs: ['5944443'] 2021-03-07T15:05:44.774 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2021-03-07T15:05:44.774 INFO:scrape: 2021-03-07T15:05:44.775 INFO:teuthology.results:Sending results to xxhdx1985126@gmail.com: Test Run: xuxuehan-2021-03-07_14:45:10-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/xuxuehan-2021-03-07_14:45:10-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/xuxuehan-2021-03-07_14:45:10-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [5944443] crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:08:44 info: http://pulpito.ceph.com/xuxuehan-2021-03-07_14:45:10-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/5944443/ log: http://qa-proxy.ceph.com/teuthology/xuxuehan-2021-03-07_14:45:10-crimson-rados:thrash-wip-crimson-interruptible-future-pipeline-parallelism-distro-basic-smithi/5944443/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0'