Found 116 jobs Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fd655b3a364] , 2: (()+0x27657e) [0x7fd655b3a57e] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-02T04:49:20.636 INFO:tasks.ceph:Scrubbing osd.6 , 2020-09-02T04:49:20.637 INFO:teuthology.orchestra.run.smithi180:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-02T04:49:43.105 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-02T04:49:43.152 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-02T04:49:43.153 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-primary-write/qa/tasks/rados.py", line 265, in thread Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fd655b3a364] , 2: (()+0x27657e) [0x7fd655b3a57e] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-02T04:49:20.636 INFO:tasks.ceph:Scrubbing osd.6 , 2020-09-02T04:49:20.637 INFO:teuthology.orchestra.run.smithi180:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-02T04:49:43.105 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-02T04:49:43.152 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-02T04:49:43.153 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-primary-write/qa/tasks/rados.py", line 265, in thread Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fd655b3a364] , 2: (()+0x27657e) [0x7fd655b3a57e] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-02T04:49:20.636 INFO:tasks.ceph:Scrubbing osd.6 , 2020-09-02T04:49:20.637 INFO:teuthology.orchestra.run.smithi180:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-02T04:49:43.105 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-02T04:49:43.152 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-02T04:49:43.153 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-primary-write/qa/tasks/rados.py", line 265, in thread Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run Ignoring malparsed backtrace: ceph version 16.0.0-4450-g8558cd72557 (8558cd725571c5c48f1a6432bb3edba52a14803c) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fd655b3a364] , 2: (()+0x27657e) [0x7fd655b3a57e] Ignoring malparsed backtrace: instructed osd(s) 5 to deep-scrub , 2020-09-02T04:49:20.636 INFO:tasks.ceph:Scrubbing osd.6 , 2020-09-02T04:49:20.637 INFO:teuthology.orchestra.run.smithi180:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.6 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-02T04:49:43.105 INFO:teuthology.orchestra.run:waiting for 300 , 2020-09-02T04:49:43.152 INFO:tasks.ceph.mgr.x:Stopped , 2020-09-02T04:49:43.153 INFO:teuthology.misc:Shutting down mon daemons... Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run , File "/home/teuthworker/src/github.com_bigwhite37_ceph_pg-primary-write/qa/tasks/rados.py", line 265, in thread Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run Found 3 distinct failure reasons Dead: b'2020-09-02T16:49:24.348 DEBUG:teuthology.exit:Finished running handlers' 3 jobs: ['5398293', '5398216', '5398225'] suites intersection: ['ceph', 'clusters/{fixed-2', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-partial-recovery}', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'msgr-failures/fastclose', 'msgr-failures/osd-delay', 'msgr/async', 'msgr/async-v1only', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/careful', 'thrashers/morepggrow', 'thrashers/pggrow', 'thrashosds-health', 'workloads/dedup_tier}', 'workloads/rados_api_tests}', 'workloads/set-chunks-read}'] Failure: Command failed (workunit test rados/test.sh) on smithi114 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8558cd725571c5c48f1a6432bb3edba52a14803c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 1 jobs: ['5398268'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-async-recovery}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-zstd', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/none', 'thrashosds-health', 'workloads/rados_api_tests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base' 1 jobs: ['5398238'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-comp-lz4', 'openstack}', 'quorum-write/3-size-2-pw-size', 'rados', 'rados:thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/none', 'thrashosds-health', 'workloads/cache-snaps-balanced}']