Found 111 jobs Ignoring malparsed backtrace: ceph version 14.2.21-328-g177c41e6ac5 (177c41e6ac5c0b7c61162a29710fbb0803876f09) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f18ad3cc3e7] , 2: (()+0x25d5af) [0x7f18ad3cc5af] Ignoring malparsed backtrace: if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2021-06-09T13:21:32.240 INFO:tasks.ceph:Scrubbing osd.1 , 2021-06-09T13:21:32.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' , 2021-06-09T13:22:20.811 INFO:teuthology.misc:Shutting down mds daemons... , 2021-06-09T13:22:20.812 INFO:teuthology.misc:Shutting down osd daemons... Ignoring malparsed backtrace: 2021-06-09T13:22:33.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2021-06-09T13:22:33.774 INFO:tasks.ceph:Compressing logs... , 2021-06-09T13:22:33.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 98, in create_image Ignoring malparsed backtrace: 2021-06-09T13:27:07.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com , 2021-06-09T13:27:07.194 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-09T13:27:07.300 INFO:teuthology.task.internal.syslog:Compressing syslogs... Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 580, in task Not-understood crash indication 2021-06-09T13:21:25.962 INFO:tasks.qemu.client.0.smithi120.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-328-g177c41e6ac5 (177c41e6ac5c0b7c61162a29710fbb0803876f09) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f18ad3cc3e7] , 2: (()+0x25d5af) [0x7f18ad3cc5af] Ignoring malparsed backtrace: if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2021-06-09T13:21:32.240 INFO:tasks.ceph:Scrubbing osd.1 , 2021-06-09T13:21:32.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' , 2021-06-09T13:22:20.811 INFO:teuthology.misc:Shutting down mds daemons... , 2021-06-09T13:22:20.812 INFO:teuthology.misc:Shutting down osd daemons... Ignoring malparsed backtrace: 2021-06-09T13:22:33.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2021-06-09T13:22:33.774 INFO:tasks.ceph:Compressing logs... , 2021-06-09T13:22:33.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 98, in create_image Ignoring malparsed backtrace: 2021-06-09T13:27:07.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com , 2021-06-09T13:27:07.194 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-09T13:27:07.300 INFO:teuthology.task.internal.syslog:Compressing syslogs... Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 580, in task Not-understood crash indication 2021-06-09T13:21:25.962 INFO:tasks.qemu.client.0.smithi120.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-328-g177c41e6ac5 (177c41e6ac5c0b7c61162a29710fbb0803876f09) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f18ad3cc3e7] , 2: (()+0x25d5af) [0x7f18ad3cc5af] Ignoring malparsed backtrace: if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) Ignoring malparsed backtrace: instructed osd(s) 0 to deep-scrub , 2021-06-09T13:21:32.240 INFO:tasks.ceph:Scrubbing osd.1 , 2021-06-09T13:21:32.241 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' , 2021-06-09T13:22:20.811 INFO:teuthology.misc:Shutting down mds daemons... , 2021-06-09T13:22:20.812 INFO:teuthology.misc:Shutting down osd daemons... Ignoring malparsed backtrace: 2021-06-09T13:22:33.672 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2021-06-09T13:22:33.774 INFO:tasks.ceph:Compressing logs... , 2021-06-09T13:22:33.775 DEBUG:teuthology.orchestra.run.smithi023:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 98, in create_image Ignoring malparsed backtrace: 2021-06-09T13:27:07.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com , 2021-06-09T13:27:07.194 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-09T13:27:07.300 INFO:teuthology.task.internal.syslog:Compressing syslogs... Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_177c41e6ac5c0b7c61162a29710fbb0803876f09/qa/tasks/qemu.py", line 580, in task Not-understood crash indication 2021-06-09T13:21:25.962 INFO:tasks.qemu.client.0.smithi120.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Found 3 distinct failure reasons Failure: Command failed on smithi078 with status 1: 'sudo yum -y install libcephfs1' 5 jobs: ['6162223', '6162257', '6162230', '6162256', '6162227'] suites intersection: ['clusters/{fixed-3', 'msgr-failures/few', 'openstack}'] suites union: ['clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'config/permit-partial-discard', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'pool/ec-data-pool', 'pool/none', 'pool/small-cache-pool', 'rbd/librbd/{cache/none', 'rbd/librbd/{cache/writeback', 'rbd/librbd/{cache/writethrough', 'rbd/qemu/{cache/writethrough', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{rhel_7}', 'supported-random-distro$/{ubuntu_16.04}', 'workloads/c_api_tests_with_journaling}', 'workloads/python_api_tests_with_journaling}', 'workloads/qemu_xfstests}'] Failure: No module named 'tasks.workunit' 3 jobs: ['6162224', '6162225', '6162222'] suites intersection: ['clusters/{fixed-3', 'msgr-failures/few', 'openstack}', 'pool/none'] suites union: ['clusters/{fixed-3', 'config/copy-on-read', 'config/none', 'config/permit-partial-discard', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'pool/none', 'rbd/librbd/{cache/none', 'rbd/librbd/{cache/writeback', 'rbd/librbd/{cache/writethrough', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_16.04}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/c_api_tests_with_journaling}', 'workloads/python_api_tests_with_journaling}'] Failure: Command failed on smithi125 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2 jobs: ['6162200', '6162271'] suites intersection: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'openstack}', 'supported-random-distro$/{centos_latest}'] suites union: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'openstack}', 'pool/replicated-data-pool', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'rbd/qemu/{cache/writethrough', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_fsstress}', 'workloads/qemu_xfstests}']