2021-06-08T23:01:54.219 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-08T23:01:54.222 INFO:teuthology.results:seed: '8023' 2021-06-08T23:19:47.872 INFO:root:teuthology version: 1.1.0-f359b10d 2021-06-08T23:19:47.908 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-06-08T23:54:50.538 INFO:teuthology.results:Tests finished! gathering results... 2021-06-08T23:54:51.754 INFO:scrape:Found 20 jobs 2021-06-08T23:54:51.928 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] 2021-06-08T23:54:51.929 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:51.929 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:51.930 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... 2021-06-08T23:54:51.956 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-06-08T23:54:51.957 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped 2021-06-08T23:54:51.957 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-06-08T23:54:51.958 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 2021-06-08T23:54:51.966 WARNING:scrape:Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:51.967 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:51.985 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] 2021-06-08T23:54:51.986 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:51.987 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:51.988 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... 2021-06-08T23:54:51.989 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-06-08T23:54:51.989 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped 2021-06-08T23:54:51.990 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-06-08T23:54:51.991 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 2021-06-08T23:54:51.998 WARNING:scrape:Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:51.999 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.017 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f2333f463e7] , 2: (()+0x25d5af) [0x7f2333f465af] 2021-06-08T23:54:52.017 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:52.018 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.019 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... 2021-06-08T23:54:52.020 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2021-06-08T23:40:50.888 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:40:50.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2021-06-08T23:54:52.020 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:17.640 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:41:17.640 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:41:17.729 INFO:tasks.ceph.osd.7:Stopped 2021-06-08T23:54:52.021 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-06-08T23:54:52.022 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:42:06.624 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:42:06.626 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/remote/smithi053 2021-06-08T23:54:52.028 WARNING:scrape:Not-understood crash indication 2021-06-08T23:40:36.629 INFO:tasks.qemu.client.0.smithi159.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.029 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.150 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fcbeff5c3e7] , 2: (()+0x25d5af) [0x7fcbeff5c5af] 2021-06-08T23:54:52.151 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:52.152 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.153 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... 2021-06-08T23:54:52.154 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:45:55.161 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:45:55.162 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 2021-06-08T23:54:52.155 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:46:42.701 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:46:42.771 INFO:tasks.ceph.osd.5:Stopped , 2021-06-08T23:46:42.771 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2021-06-08T23:54:52.155 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:52.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:47:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-06-08T23:47:58.946 DEBUG:teuthology.orchestra.run.smithi068:> test -e /home/ubuntu/cephtest/archive/coredump , 2021-06-08T23:47:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-06-08T23:54:52.163 WARNING:scrape:Not-understood crash indication 2021-06-08T23:45:39.639 INFO:tasks.qemu.client.0.smithi165.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.164 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.323 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] 2021-06-08T23:54:52.324 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.339 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:52.340 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:52.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped 2021-06-08T23:54:52.341 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): 2021-06-08T23:54:52.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-06-08T23:54:52.343 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:52.350 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.351 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.371 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] 2021-06-08T23:54:52.372 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.374 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:52.375 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:52.375 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped 2021-06-08T23:54:52.376 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): 2021-06-08T23:54:52.377 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-06-08T23:54:52.377 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:52.384 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.385 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.404 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] 2021-06-08T23:54:52.405 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.406 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:52.407 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:52.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped 2021-06-08T23:54:52.408 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): 2021-06-08T23:54:52.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-06-08T23:54:52.410 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:52.417 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.418 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.438 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f3b93dbf3e7] , 2: (()+0x25d5af) [0x7f3b93dbf5af] 2021-06-08T23:54:52.439 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.440 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:52.441 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:52.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:12.132 DEBUG:tasks.ceph.mon.a:waiting for process to exit , 2021-06-08T23:40:12.133 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:12.153 INFO:tasks.ceph.mon.a:Stopped 2021-06-08T23:54:52.442 WARNING:scrape:Ignoring malparsed backtrace: yield , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 47, in nested , if exit(*exc): 2021-06-08T23:54:52.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:41:35.780 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:41:35.868 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com , 2021-06-08T23:41:35.869 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-06-08T23:54:52.443 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:52.451 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:15.655 INFO:tasks.qemu.client.0.smithi187.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.452 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.474 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f811becc3e7] , 2: (()+0x25d5af) [0x7f811becc5af] 2021-06-08T23:54:52.476 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-06-08T23:54:52.476 WARNING:scrape:Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:31:37.452 ERROR:teuthology.contextutil:Saw exception from nested tasks 2021-06-08T23:54:52.477 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:31:37.547 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:31:37.548 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:31:37.549 INFO:teuthology.orchestra.run:waiting for 300 2021-06-08T23:54:52.477 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:32:24.925 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi123/log , 2021-06-08T23:32:24.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:32:30.803 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi181/log 2021-06-08T23:54:52.478 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:32:31.153 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:32:31.189 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:32:31.190 DEBUG:teuthology.orchestra.run.smithi058:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-06-08T23:54:52.479 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:52.479 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393750 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393753 60 -rw-rw-rw- 1 ubuntu ubuntu 58935 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2021-06-08T23:54:52.487 WARNING:scrape:Not-understood crash indication 2021-06-08T23:31:03.345 INFO:tasks.qemu.client.0.smithi181.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.488 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.509 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f811becc3e7] , 2: (()+0x25d5af) [0x7f811becc5af] 2021-06-08T23:54:52.511 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-06-08T23:54:52.511 WARNING:scrape:Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:31:37.452 ERROR:teuthology.contextutil:Saw exception from nested tasks 2021-06-08T23:54:52.512 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:31:37.547 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:31:37.548 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:31:37.549 INFO:teuthology.orchestra.run:waiting for 300 2021-06-08T23:54:52.513 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:32:24.925 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi123/log , 2021-06-08T23:32:24.927 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:32:30.803 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/remote/smithi181/log 2021-06-08T23:54:52.513 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:32:31.153 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:32:31.189 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:32:31.190 DEBUG:teuthology.orchestra.run.smithi058:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-06-08T23:54:52.514 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:52.515 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393750 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393752 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog , 2021-06-08T23:37:18.613 INFO:teuthology.orchestra.run.smithi058.stdout:393753 60 -rw-rw-rw- 1 ubuntu ubuntu 58935 Jun 8 23:32 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2021-06-08T23:54:52.522 WARNING:scrape:Not-understood crash indication 2021-06-08T23:31:03.345 INFO:tasks.qemu.client.0.smithi181.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.523 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.679 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f88b919e3e7] , 2: (()+0x25d5af) [0x7f88b919e5af] 2021-06-08T23:54:52.755 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-06-08T23:54:52.756 WARNING:scrape:Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:35:59.108 ERROR:teuthology.contextutil:Saw exception from nested tasks 2021-06-08T23:54:52.756 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:35:59.213 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:35:59.214 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:35:59.215 INFO:teuthology.orchestra.run:waiting for 300 2021-06-08T23:54:52.757 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:38.812 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi105/log , 2021-06-08T23:36:38.814 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:36:40.941 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi183/log 2021-06-08T23:54:52.758 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:41.297 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:36:41.307 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:36:41.308 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-06-08T23:54:52.759 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:52.759 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393812 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393863 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:394037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive/syslog 2021-06-08T23:54:52.767 WARNING:scrape:Not-understood crash indication 2021-06-08T23:35:24.201 INFO:tasks.qemu.client.0.smithi183.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.768 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.790 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f88b919e3e7] , 2: (()+0x25d5af) [0x7f88b919e5af] 2021-06-08T23:54:52.792 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-06-08T23:54:52.792 WARNING:scrape:Ignoring malparsed backtrace: mon.b: injectargs:mon_health_to_clog = 'false' , mon.c: injectargs:mon_health_to_clog = 'false' , 2021-06-08T23:35:59.108 ERROR:teuthology.contextutil:Saw exception from nested tasks 2021-06-08T23:54:52.793 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:35:59.213 INFO:tasks.ceph.osd.1:Stopped , 2021-06-08T23:35:59.214 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-06-08T23:35:59.215 INFO:teuthology.orchestra.run:waiting for 300 2021-06-08T23:54:52.794 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:38.812 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi105/log , 2021-06-08T23:36:38.814 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-06-08T23:36:40.941 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/remote/smithi183/log 2021-06-08T23:54:52.794 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:41.297 DEBUG:teuthology.run_tasks:Unwinding manager clock , 2021-06-08T23:36:41.307 INFO:teuthology.task.clock:Checking final clock skew... , 2021-06-08T23:36:41.308 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-06-08T23:54:52.795 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:52.796 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393812 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:393863 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive , 2021-06-08T23:42:11.783 INFO:teuthology.orchestra.run.smithi061.stdout:394037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 8 23:36 /home/ubuntu/cephtest/archive/syslog 2021-06-08T23:54:52.805 WARNING:scrape:Not-understood crash indication 2021-06-08T23:35:24.201 INFO:tasks.qemu.client.0.smithi183.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.806 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:52.842 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f27a195e3e7] , 2: (()+0x25d5af) [0x7f27a195e5af] 2021-06-08T23:54:52.843 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:52.844 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:52.845 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 62% complete... , Removing image: 63% complete... , Removing image: 64% complete... 2021-06-08T23:54:52.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:29:15.907 INFO:tasks.ceph:Scrubbing osd.4 , 2021-06-08T23:29:15.908 DEBUG:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , Set osd_debug_deep_scrub_sleep to 0 2021-06-08T23:54:52.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:29:41.604 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:29:41.684 INFO:tasks.ceph.osd.7:Stopped , 2021-06-08T23:29:41.685 INFO:teuthology.misc:Shutting down mgr daemons... 2021-06-08T23:54:52.848 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/contextutil.py", line 55, in nested 2021-06-08T23:54:52.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:31:17.990 INFO:teuthology.task.internal:Transferring archived files... , 2021-06-08T23:31:17.992 DEBUG:teuthology.misc:Transferring archived files from smithi081:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161219/remote/smithi081 , 2021-06-08T23:31:17.992 DEBUG:teuthology.orchestra.run.smithi081:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-06-08T23:54:52.858 WARNING:scrape:Not-understood crash indication 2021-06-08T23:29:02.845 INFO:tasks.qemu.client.0.smithi094.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:52.859 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.000 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f8a123bf3e7] , 2: (()+0x25d5af) [0x7f8a123bf5af] 2021-06-08T23:54:53.002 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:53.003 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:53.004 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait 2021-06-08T23:54:53.005 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:52.526 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:52.583 INFO:tasks.ceph.mon.c:Stopped , 2021-06-08T23:40:52.584 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-06-08T23:54:53.005 WARNING:scrape:Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) 2021-06-08T23:54:53.006 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:00.771 DEBUG:teuthology.orchestra.run.smithi201:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:42:00.876 INFO:teuthology.task.internal.syslog:Compressing syslogs... , 2021-06-08T23:42:00.877 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-06-08T23:54:53.007 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:53.014 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:55.679 INFO:tasks.qemu.client.0.smithi201.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:53.015 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.035 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f8a123bf3e7] , 2: (()+0x25d5af) [0x7f8a123bf5af] 2021-06-08T23:54:53.035 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:53.037 WARNING:scrape:Ignoring malparsed backtrace: name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run , r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2021-06-08T23:54:53.038 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 423, in run_qemu , run.wait(procs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 479, in wait 2021-06-08T23:54:53.038 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:40:52.526 INFO:teuthology.orchestra.run:waiting for 300 , 2021-06-08T23:40:52.583 INFO:tasks.ceph.mon.c:Stopped , 2021-06-08T23:40:52.584 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-06-08T23:54:53.039 WARNING:scrape:Ignoring malparsed backtrace: self.gen.throw(type, value, traceback) , File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 173, in clone_image , remote.run(args=args) 2021-06-08T23:54:53.040 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:42:00.771 DEBUG:teuthology.orchestra.run.smithi201:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 , 2021-06-08T23:42:00.876 INFO:teuthology.task.internal.syslog:Compressing syslogs... , 2021-06-08T23:42:00.877 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-06-08T23:54:53.040 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2021-06-08T23:54:53.047 WARNING:scrape:Not-understood crash indication 2021-06-08T23:39:55.679 INFO:tasks.qemu.client.0.smithi201.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:53.048 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.190 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f7f141bb3e7] , 2: (()+0x25d5af) [0x7f7f141bb5af] 2021-06-08T23:54:53.190 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 83% complete... , Removing image: 84% complete... , Removing image: 85% complete... 2021-06-08T23:54:53.191 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:53.192 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 60% complete... , Removing image: 61% complete... , Removing image: 62% complete... 2021-06-08T23:54:53.193 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:32:38.576 DEBUG:teuthology.orchestra.run.smithi085:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2021-06-08T23:32:39.067 INFO:tasks.ceph:Scrubbing osd.4 2021-06-08T23:54:53.193 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:33:05.030 INFO:tasks.ceph.osd.6:Stopped , 2021-06-08T23:33:05.031 DEBUG:tasks.ceph.osd.7:waiting for process to exit , 2021-06-08T23:33:05.031 INFO:teuthology.orchestra.run:waiting for 300 2021-06-08T23:54:53.194 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/qemu.py", line 580, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-06-08T23:54:53.195 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:33:52.128 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi168.front.sepia.ceph.com, flagging run as failed , 2021-06-08T23:33:52.129 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-06-08T23:33:52.143 INFO:teuthology.task.internal:Transferring archived files... 2021-06-08T23:54:53.202 WARNING:scrape:Not-understood crash indication 2021-06-08T23:32:26.024 INFO:tasks.qemu.client.0.smithi168.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:53.203 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.326 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fa1f91843e7] , 2: (()+0x25d5af) [0x7fa1f91845af] 2021-06-08T23:54:53.327 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:53.335 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 98, in create_image , name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run 2021-06-08T23:54:53.336 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:53.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:40.006 INFO:tasks.ceph:Checking cluster log for badness... , 2021-06-08T23:36:40.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 , 2021-06-08T23:36:40.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com 2021-06-08T23:54:53.337 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait 2021-06-08T23:54:53.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:37:42.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-06-08T23:37:42.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-06-08T23:37:42.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-06-08T23:54:53.338 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2021-06-08T23:54:53.346 WARNING:scrape:Not-understood crash indication 2021-06-08T23:36:03.958 INFO:tasks.qemu.client.0.smithi161.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:53.347 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.367 WARNING:scrape:Ignoring malparsed backtrace: ceph version 14.2.21-341-gb0721aafdd1 (b0721aafdd1f6c07434cda2c9efedd01dfd96bcb) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fa1f91843e7] , 2: (()+0x25d5af) [0x7fa1f91845af] 2021-06-08T23:54:53.367 WARNING:scrape:Ignoring malparsed backtrace: Removing image: 75% complete... , Removing image: 76% complete... , Removing image: 77% complete... 2021-06-08T23:54:53.369 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph-c_b0721aafdd1f6c07434cda2c9efedd01dfd96bcb/qa/tasks/rbd.py", line 98, in create_image , name, , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/remote.py", line 509, in run 2021-06-08T23:54:53.369 WARNING:scrape:Ignoring malparsed backtrace: self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 183, in _raise_for_status , node=self.hostname, label=self.label 2021-06-08T23:54:53.370 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:36:40.006 INFO:tasks.ceph:Checking cluster log for badness... , 2021-06-08T23:36:40.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | head -n 1 , 2021-06-08T23:36:40.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com 2021-06-08T23:54:53.371 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 455, in run , r.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_f359b10daba6e0103d42ccfc021bc797f3cd7edc/teuthology/orchestra/run.py", line 161, in wait 2021-06-08T23:54:53.371 WARNING:scrape:Ignoring malparsed backtrace: 2021-06-08T23:37:42.946 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo , 2021-06-08T23:37:42.957 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2021-06-08T23:37:42.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-06-08T23:54:53.372 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ , self.gen.throw(type, value, traceback) 2021-06-08T23:54:53.380 WARNING:scrape:Not-understood crash indication 2021-06-08T23:36:03.958 INFO:tasks.qemu.client.0.smithi161.stderr:daemon-helper: command crashed with signal 6 2021-06-08T23:54:53.381 WARNING:scrape:Not-understood crash indication 2021-06-08T23:54:53.382 INFO:scrape:Found 2 distinct failure reasons 2021-06-08T23:54:53.382 INFO:scrape:Failure: Command failed on smithi159 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' 2021-06-08T23:54:53.383 INFO:scrape:4 jobs: ['6161234', '6161237', '6161219', '6161222'] 2021-06-08T23:54:53.383 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] 2021-06-08T23:54:53.384 INFO:scrape:suites union: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] 2021-06-08T23:54:53.384 INFO:scrape: 2021-06-08T23:54:53.385 INFO:scrape:Failure: Command failed on smithi187 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' 2021-06-08T23:54:53.385 INFO:scrape:5 jobs: ['6161229', '6161221', '6161226', '6161230', '6161225'] 2021-06-08T23:54:53.386 INFO:scrape:suites intersection: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] 2021-06-08T23:54:53.386 INFO:scrape:suites union: ['clusters/{fixed-3', 'features/journaling', 'msgr-failures/few', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/qemu/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_bonnie}'] 2021-06-08T23:54:53.387 INFO:scrape: 2021-06-08T23:54:53.388 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [6161219] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:18:51 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161219/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161219/ Command failed on smithi094 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' [6161221] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:16:17 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161221/ Command failed on smithi181 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' [6161222] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:15:41 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161222/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161222/ Command failed on smithi168 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' [6161225] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:18:43 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161225/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161225/ Command failed on smithi161 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' [6161226] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:17:11 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161226/ Command failed on smithi183 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' [6161229] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:19:16 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161229/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161229/ Command failed on smithi187 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' [6161230] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:19:55 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161230/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161230/ Command failed on smithi201 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.1' [6161234] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:16:31 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161234/ Command failed on smithi159 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' [6161237] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} ----------------------------------------------------------------- time: 00:18:03 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161237/ log: http://qa-proxy.ceph.com/teuthology/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161237/ Command failed on smithi165 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term /usr/libexec/qemu-kvm -enable-kvm -nographic -cpu host -smp 1 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback' Pass ================================================================= [6161218] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:24:13 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161218/ [6161220] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:24:29 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161220/ [6161223] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:24:49 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161223/ [6161224] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:25:01 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161224/ [6161227] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:27:56 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161227/ [6161228] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:27:51 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161228/ [6161231] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:26:24 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161231/ [6161232] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:26:17 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161232/ [6161233] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:25:29 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161233/ [6161235] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:24:09 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161235/ [6161236] rbd/qemu/{cache/writeback clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/small-cache-pool supported-random-distro$/{centos_latest} workloads/qemu_bonnie} time: 00:24:33 info: http://pulpito.ceph.com/dis-2021-06-08_23:01:13-rbd-wip-dis-testing-nautilus-distro-basic-smithi/6161236/