Found 20 jobs Ignoring malparsed backtrace: ceph version 16.0.0-5799-g536b9bcf (536b9bcf1190b7df8e77e7099eafbf937d1d7001) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f94e3f004f8] , 2: () Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2020-09-30T13:37:15.195 INFO:tasks.ceph:Scrubbing osd.2 , 2020-09-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-30T13:37:38.566 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-09-30T13:37:58.567 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-30T13:37:58.864 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-09-30T13:40:22.792 INFO:teuthology.misc:Shutting down mon daemons... , 2020-09-30T13:40:22.794 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2020-09-30T13:40:22.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv02.front.sepia.ceph.com 2 9 377 169 +225us[ +173us] +/- 42ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 92 -826us[ -826us] +/- 70ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^- hv04.front.sepia.ceph.com 3 7 377 34 +725us[ +725us] +/- 74ms Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status , node=self.hostname, label=self.label Ignoring malparsed backtrace: ceph version 16.0.0-5799-g536b9bcf (536b9bcf1190b7df8e77e7099eafbf937d1d7001) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f94e3f004f8] , 2: () Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2020-09-30T13:37:15.195 INFO:tasks.ceph:Scrubbing osd.2 , 2020-09-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-30T13:37:38.566 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-09-30T13:37:58.567 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-30T13:37:58.864 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-09-30T13:40:22.792 INFO:teuthology.misc:Shutting down mon daemons... , 2020-09-30T13:40:22.794 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2020-09-30T13:40:22.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv02.front.sepia.ceph.com 2 9 377 169 +225us[ +173us] +/- 42ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 92 -826us[ -826us] +/- 70ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^- hv04.front.sepia.ceph.com 3 7 377 34 +725us[ +725us] +/- 74ms Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status , node=self.hostname, label=self.label Ignoring malparsed backtrace: ceph version 16.0.0-5799-g536b9bcf (536b9bcf1190b7df8e77e7099eafbf937d1d7001) pacific (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f94e3f004f8] , 2: () Ignoring malparsed backtrace: instructed osd(s) 1 to deep-scrub , 2020-09-30T13:37:15.195 INFO:tasks.ceph:Scrubbing osd.2 , 2020-09-30T13:37:15.195 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early tell osd.2 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2020-09-30T13:37:38.566 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-09-30T13:37:58.567 INFO:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json , 2020-09-30T13:37:58.864 INFO:teuthology.orchestra.run.smithi086.stdout: Ignoring malparsed backtrace: 2020-09-30T13:40:22.792 INFO:teuthology.misc:Shutting down mon daemons... , 2020-09-30T13:40:22.794 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2020-09-30T13:40:22.794 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv02.front.sepia.ceph.com 2 9 377 169 +225us[ +173us] +/- 42ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 92 -826us[ -826us] +/- 70ms , 2020-09-30T13:44:24.589 INFO:teuthology.orchestra.run.smithi018.stdout:^- hv04.front.sepia.ceph.com 3 7 377 34 +725us[ +725us] +/- 74ms Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 182, in _raise_for_status , node=self.hostname, label=self.label Found 2 distinct failure reasons Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi142 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6ae0efe84c112479039110d140f7deba873f788 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 RBD_FEATURES=125 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 8 jobs: ['5481240', '5481233', '5481248', '5481236', '5481242', '5481245', '5481239', '5481230'] suites intersection: ['openstack}'] suites union: ['clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'config/permit-partial-discard', 'min-compat-client/default', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'pool/ceph_and_immutable_object_cache', 'pool/none', 'pool/replicated-data-pool', 'rbd/immutable-object-cache/{clusters/{fix-2', 'rbd/librbd/{cache/writearound', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_defaults}', 'workloads/c_api_tests_with_journaling}'] Failure: "2020-09-30T12:23:35.837273+0000 mon.a (mon.0) 1880 : cluster [WRN] Health check failed: Degraded data redundancy: 2/20 objects degraded (10.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 1 jobs: ['5481243'] suites: ['clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writeback', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}']