Found 20 jobs Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fcbeff5c3e7] , 2: (()+0x25d5af) [0x7fcbeff5c5af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... Ignoring malparsed backtrace: 2021-06-08T23:45:55.161 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:45:55.162 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 Ignoring malparsed backtrace: 2021-06-08T23:46:42.701 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:46:42.771 INFO:tasks.ceph.osd.5:Stopped , 2021-06-08T23:46:42.771 DEBUG:tasks.ceph.osd.6:waiting for process to exit Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:47:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-06-08T23:47:58.946 DEBUG:teuthology.orchestra.run.smithi068:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-06-08T23:47:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 Not-understood crash indication 2021-06-08T23:45:39.639 INFO:tasks.qemu.client.0.smithi165.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f811becc3e7] , 2: (()+0x25d5af) [0x7f811becc5af] Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:31:37.452 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-06-08T23:31:37.547 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:31:37.548 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:31:37.549 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-06-08T23:32:24.925 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi123/log , 2021-06-08T23:32:24.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:32:30.803 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi181/log Ignoring malparsed backtrace: 2021-06-08T23:32:31.153 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:32:31.189 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:32:31.190 DEBUG:teuthology.orchestra.run.smithi058:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Ignoring malparsed backtrace: 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393750 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393753 60 -rw-rw-rw- 1 ubuntu ubuntu 58935 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz Not-understood crash indication 2021-06-08T23:31:03.345 INFO:tasks.qemu.client.0.smithi181.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f811becc3e7] , 2: (()+0x25d5af) [0x7f811becc5af] Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:31:37.452 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-06-08T23:31:37.547 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:31:37.548 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:31:37.549 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-06-08T23:32:24.925 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi123/log , 2021-06-08T23:32:24.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:32:30.803 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi181/log Ignoring malparsed backtrace: 2021-06-08T23:32:31.153 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:32:31.189 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:32:31.190 DEBUG:teuthology.orchestra.run.smithi058:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Ignoring malparsed backtrace: 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393750 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393753 60 -rw-rw-rw- 1 ubuntu ubuntu 58935 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz Not-understood crash indication 2021-06-08T23:31:03.345 INFO:tasks.qemu.client.0.smithi181.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f88b919e3e7] , 2: (()+0x25d5af) [0x7f88b919e5af] Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:35:59.108 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-06-08T23:35:59.213 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:35:59.214 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:35:59.215 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-06-08T23:36:38.812 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi105/log , 2021-06-08T23:36:38.814 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:36:40.941 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi183/log Ignoring malparsed backtrace: 2021-06-08T23:36:41.297 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:36:41.307 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:36:41.308 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Ignoring malparsed backtrace: 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393812 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393863 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:394037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive/syslog Not-understood crash indication 2021-06-08T23:35:24.201 INFO:tasks.qemu.client.0.smithi183.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f88b919e3e7] , 2: (()+0x25d5af) [0x7f88b919e5af] Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:35:59.108 ERROR:teuthology.contextutil:Saw exception from nested tasks Ignoring malparsed backtrace: 2021-06-08T23:35:59.213 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:35:59.214 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:35:59.215 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: 2021-06-08T23:36:38.812 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi105/log , 2021-06-08T23:36:38.814 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:36:40.941 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi183/log Ignoring malparsed backtrace: 2021-06-08T23:36:41.297 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:36:41.307 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:36:41.308 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Ignoring malparsed backtrace: 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393812 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393863 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:394037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive/syslog Not-understood crash indication 2021-06-08T23:35:24.201 INFO:tasks.qemu.client.0.smithi183.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f27a195e3e7] , 2: (()+0x25d5af) [0x7f27a195e5af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... Ignoring malparsed backtrace: 2021-06-08T23:29:15.907 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:29:15.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 Ignoring malparsed backtrace: 2021-06-08T23:29:41.604 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:29:41.684 INFO:tasks.ceph.osd.7:Stopped , 2021-06-08T23:29:41.685 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 55, in nested Ignoring malparsed backtrace: 2021-06-08T23:31:17.990 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:31:17.992 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161219/remote/smithi081 , 2021-06-08T23:31:17.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . Not-understood crash indication 2021-06-08T23:29:02.845 INFO:tasks.qemu.client.0.smithi094.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f8a123bf3e7] , 2: (()+0x25d5af) [0x7f8a123bf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait Ignoring malparsed backtrace: 2021-06-08T23:40:52.526 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:52.583 INFO:tasks.ceph.mon.c:Stopped , 2021-06-08T23:40:52.584 DEBUG:tasks.ceph.mon.b:waiting for process to exit Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) Ignoring malparsed backtrace: 2021-06-08T23:42:00.771 DEBUG:teuthology.orchestra.run.smithi201:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:42:00.876 INFO:teuthology.task.internal.syslog:Compressing syslogs... , 2021-06-08T23:42:00.877 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Not-understood crash indication 2021-06-08T23:39:55.679 INFO:tasks.qemu.client.0.smithi201.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f8a123bf3e7] , 2: (()+0x25d5af) [0x7f8a123bf5af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait Ignoring malparsed backtrace: 2021-06-08T23:40:52.526 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:52.583 INFO:tasks.ceph.mon.c:Stopped , 2021-06-08T23:40:52.584 DEBUG:tasks.ceph.mon.b:waiting for process to exit Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) Ignoring malparsed backtrace: 2021-06-08T23:42:00.771 DEBUG:teuthology.orchestra.run.smithi201:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:42:00.876 INFO:teuthology.task.internal.syslog:Compressing syslogs... , 2021-06-08T23:42:00.877 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , Not-understood crash indication 2021-06-08T23:39:55.679 INFO:tasks.qemu.client.0.smithi201.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f7f141bb3e7] , 2: (()+0x25d5af) [0x7f7f141bb5af] Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: Removing image: 60% complete... , Removing image: 61% complete... , Removing image: 62% complete... Ignoring malparsed backtrace: 2021-06-08T23:32:38.576 DEBUG:teuthology.orchestra.run.smithi085:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2021-06-08T23:32:39.067 INFO:tasks.ceph:Scrubbing osd.4 Ignoring malparsed backtrace: 2021-06-08T23:33:05.030 INFO:tasks.ceph.osd.6:Stopped , 2021-06-08T23:33:05.031 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:33:05.031 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 580, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: 2021-06-08T23:33:52.128 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi168.front.sepia.ceph.com, flagging run as failed , 2021-06-08T23:33:52.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:33:52.143 INFO:teuthology.task.internal:Transferring archived files... Not-understood crash indication 2021-06-08T23:32:26.024 INFO:tasks.qemu.client.0.smithi168.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fa1f91843e7] , 2: (()+0x25d5af) [0x7fa1f91845af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 98, in create_image , name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Ignoring malparsed backtrace: 2021-06-08T23:36:40.006 INFO:tasks.ceph:Checking cluster log for badness... , 2021-06-08T23:36:40.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 , 2021-06-08T23:36:40.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait Ignoring malparsed backtrace: 2021-06-08T23:37:42.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-06-08T23:37:42.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-06-08T23:37:42.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) Not-understood crash indication 2021-06-08T23:36:03.958 INFO:tasks.qemu.client.0.smithi161.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fa1f91843e7] , 2: (()+0x25d5af) [0x7fa1f91845af] Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 98, in create_image , name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label Ignoring malparsed backtrace: 2021-06-08T23:36:40.006 INFO:tasks.ceph:Checking cluster log for badness... , 2021-06-08T23:36:40.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 , 2021-06-08T23:36:40.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait Ignoring malparsed backtrace: 2021-06-08T23:37:42.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-06-08T23:37:42.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-06-08T23:37:42.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) Not-understood crash indication 2021-06-08T23:36:03.958 INFO:tasks.qemu.client.0.smithi161.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Found 2 distinct failure reasons Failure: Command failed on smithi159 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' 4 jobs: ['6161234', '6161237', '6161219', '6161222'] suites intersection: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] suites union: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] Failure: Command failed on smithi187 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' 5 jobs: ['6161229', '6161221', '6161226', '6161230', '6161225'] suites intersection: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] suites union: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}']