Found 116 jobs Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f69ac5f2aa4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x555c04090746] Ignoring malparsed backtrace: 2020-09-03T04:02:13.624 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 4 , 2020-09-03T04:02:13.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:13.829+0000 7f8daadd3700 -1 osd.2 38 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4506.0:2076 4.e 4:71b0e392:::smithi18336296-145:head [assert-version v79,set-chunk in=390b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e24) Ignoring malparsed backtrace: 2020-09-03T04:02:35.596 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-03T04:02:35.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:35.877 INFO:teuthology.orchestra.run.smithi183.stdout: Ignoring malparsed backtrace: 2020-09-03T04:02:56.859 DEBUG:tasks.ceph.osd.6:waiting for process to exit , 2020-09-03T04:02:56.860 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-03T04:02:56.937 INFO:tasks.ceph.osd.6:Stopped Ignoring malparsed backtrace: 2020-09-03T04:04:28.982 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested Ignoring malparsed backtrace: 2020-09-03T04:04:52.575 INFO:teuthology.orchestra.run.smithi183:> sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.56NJSDAW7J , cp: cannot stat '/usr/bin/ceph_test_rados.debug': No such file or directory , 2020-09-03T04:04:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f69ac5f2aa4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x555c04090746] Ignoring malparsed backtrace: 2020-09-03T04:02:13.624 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 4 , 2020-09-03T04:02:13.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:13.829+0000 7f8daadd3700 -1 osd.2 38 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4506.0:2076 4.e 4:71b0e392:::smithi18336296-145:head [assert-version v79,set-chunk in=390b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e24) Ignoring malparsed backtrace: 2020-09-03T04:02:35.596 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-03T04:02:35.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:35.877 INFO:teuthology.orchestra.run.smithi183.stdout: Ignoring malparsed backtrace: 2020-09-03T04:02:56.859 DEBUG:tasks.ceph.osd.6:waiting for process to exit , 2020-09-03T04:02:56.860 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-03T04:02:56.937 INFO:tasks.ceph.osd.6:Stopped Ignoring malparsed backtrace: 2020-09-03T04:04:28.982 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested Ignoring malparsed backtrace: 2020-09-03T04:04:52.575 INFO:teuthology.orchestra.run.smithi183:> sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.56NJSDAW7J , cp: cannot stat '/usr/bin/ceph_test_rados.debug': No such file or directory , 2020-09-03T04:04:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x7f69ac5f2aa4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x666) [0x555c04090746] Ignoring malparsed backtrace: 2020-09-03T04:02:13.624 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early osd deep-scrub 4 , 2020-09-03T04:02:13.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:13.829+0000 7f8daadd3700 -1 osd.2 38 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4506.0:2076 4.e 4:71b0e392:::smithi18336296-145:head [assert-version v79,set-chunk in=390b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e24) Ignoring malparsed backtrace: 2020-09-03T04:02:35.596 INFO:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-03T04:02:35.721+0000 7f489b54f700 -1 osd.0 38 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4506.0:3973 4.c 4:33700d88:::smithi18336296-242:head [assert-version v206,set-chunk in=89b] snapc 0=[] RETRY=1 ondisk+retry+write+rwordered+known_if_redirected e38) , 2020-09-03T04:02:35.877 INFO:teuthology.orchestra.run.smithi183.stdout: Ignoring malparsed backtrace: 2020-09-03T04:02:56.859 DEBUG:tasks.ceph.osd.6:waiting for process to exit , 2020-09-03T04:02:56.860 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-03T04:02:56.937 INFO:tasks.ceph.osd.6:Stopped Ignoring malparsed backtrace: 2020-09-03T04:04:28.982 ERROR:teuthology.contextutil:Saw exception from nested tasks , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 33, in nested Ignoring malparsed backtrace: 2020-09-03T04:04:52.575 INFO:teuthology.orchestra.run.smithi183:> sudo cp /usr/bin/ceph_test_rados.debug /tmp/tmp.56NJSDAW7J , cp: cannot stat '/usr/bin/ceph_test_rados.debug': No such file or directory , 2020-09-03T04:04:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 Found 3 distinct failure reasons Dead: b'2020-09-03T14:49:59.822 DEBUG:teuthology.exit:Finished running handlers' 2 jobs: ['5401394', '5401376'] suites intersection: ['1-pg-log-overrides/short_pg_log', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'thrashosds-health'] suites union: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-recovery}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'msgr-failures/fastclose', 'msgr/async-v1only', 'objectstore/bluestore-comp-zlib', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/rados_api_tests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --low_tier_pool low_tier --max-ops 4000 --objects 300 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 10 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['5401383'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/none', 'thrashosds-health', 'workloads/set-chunks-read}'] Failure: Command failed (workunit test rados/test.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae1b70329677647b1ea7a1fbd98d84f65ad90a29 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2 jobs: ['5401301', '5401326'] suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'thrashosds-health', 'workloads/rados_api_tests}'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr/async-v1only', 'msgr/async-v2only', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}']