2023-03-19T18:56:34.727 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3
2023-03-19T18:56:35.481 INFO:teuthology.results:seed: '1856'
2023-03-19T19:54:22.758 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3
2023-03-19T19:54:23.997 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-03-20T07:14:14.405 INFO:teuthology.results:Tests finished! gathering results...
2023-03-20T07:14:16.730 INFO:scrape:Found 18 jobs
2023-03-20T07:14:18.908 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7ff3a8df57ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff3a8df5a2a]

2023-03-20T07:14:19.733 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.0 to 1 (10000)
, 2023-03-19T20:35:47.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1
, 2023-03-19T20:35:48.099+0000 7fea34097700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:19.733 WARNING:scrape:Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph_manager.py", line 1588, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

2023-03-20T07:14:20.200 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:29:43.917 INFO:tasks.ceph.osd.0:Stopped
, 2023-03-19T21:29:44.761 DEBUG:tasks.ceph.osd.1:waiting for process to exit
, 2023-03-19T21:29:44.761 INFO:teuthology.orchestra.run:waiting for 300

2023-03-20T07:14:20.200 WARNING:scrape:Ignoring malparsed backtrace:     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 188, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-03-20T07:14:20.200 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:21.292 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/rados.py", line 286, in task
,     running.get()

2023-03-20T07:14:21.305 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7ff3a8df57ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff3a8df5a2a]

2023-03-20T07:14:22.920 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.0 to 1 (10000)
, 2023-03-19T20:35:47.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1
, 2023-03-19T20:35:48.099+0000 7fea34097700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:24.147 WARNING:scrape:Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph_manager.py", line 1588, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

2023-03-20T07:14:24.148 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:29:43.917 INFO:tasks.ceph.osd.0:Stopped
, 2023-03-19T21:29:44.761 DEBUG:tasks.ceph.osd.1:waiting for process to exit
, 2023-03-19T21:29:44.761 INFO:teuthology.orchestra.run:waiting for 300

2023-03-20T07:14:24.148 WARNING:scrape:Ignoring malparsed backtrace:     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 188, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-03-20T07:14:24.148 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:24.148 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/rados.py", line 286, in task
,     running.get()

2023-03-20T07:14:24.161 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7ff3a8df57ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff3a8df5a2a]

2023-03-20T07:14:25.194 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.0 to 1 (10000)
, 2023-03-19T20:35:47.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1
, 2023-03-19T20:35:48.099+0000 7fea34097700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:25.194 WARNING:scrape:Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph_manager.py", line 1588, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

2023-03-20T07:14:25.195 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:29:43.917 INFO:tasks.ceph.osd.0:Stopped
, 2023-03-19T21:29:44.761 DEBUG:tasks.ceph.osd.1:waiting for process to exit
, 2023-03-19T21:29:44.761 INFO:teuthology.orchestra.run:waiting for 300

2023-03-20T07:14:25.195 WARNING:scrape:Ignoring malparsed backtrace:     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 188, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-03-20T07:14:25.195 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:26.950 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/rados.py", line 286, in task
,     running.get()

2023-03-20T07:14:26.963 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7ff3a8df57ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff3a8df5a2a]

2023-03-20T07:14:28.857 WARNING:scrape:Ignoring malparsed backtrace: reweighted osd.0 to 1 (10000)
, 2023-03-19T20:35:47.966 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1
, 2023-03-19T20:35:48.099+0000 7fea34097700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:30.477 WARNING:scrape:Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph_manager.py", line 1588, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

2023-03-20T07:14:32.569 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:29:43.917 INFO:tasks.ceph.osd.0:Stopped
, 2023-03-19T21:29:44.761 DEBUG:tasks.ceph.osd.1:waiting for process to exit
, 2023-03-19T21:29:44.761 INFO:teuthology.orchestra.run:waiting for 300

2023-03-20T07:14:32.569 WARNING:scrape:Ignoring malparsed backtrace:     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/run_tasks.py", line 188, in run_tasks
,     suppress = manager.__exit__(*exc_info)

2023-03-20T07:14:32.569 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:32.569 WARNING:scrape:Ignoring malparsed backtrace:     next(self.gen)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/rados.py", line 286, in task
,     running.get()

2023-03-20T07:14:33.723 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7fe0bceb67ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe0bceb6a2a]

2023-03-20T07:14:34.305 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:21:51.228+0000 7f4557104700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:21:51.237+0000 7f4557104700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, reweighted osd.1 to 1 (10000)

2023-03-20T07:14:34.305 WARNING:scrape:Ignoring malparsed backtrace:     running.get()
,   File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
,   File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception

2023-03-20T07:14:34.305 WARNING:scrape:Ignoring malparsed backtrace:     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
,     next(self.gen)

2023-03-20T07:14:34.305 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:34.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:15:08.583 INFO:tasks.ceph:Archiving crash dumps...
, 2023-03-19T21:15:08.585 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-03-19_18:47:10-crimson-rados-wip-matanb-crimson-only-testing-19.3-v2-distro-crimson-smithi/7213768/remote/smithi149/crash
, 2023-03-19T21:15:12.589 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/crash -- .

2023-03-20T07:14:34.306 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:16:26.837 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump
, 2023-03-19T21:16:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2023-03-19T21:16:27.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive

2023-03-20T07:14:34.318 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7fe0bceb67ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fe0bceb6a2a]

2023-03-20T07:14:35.824 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:21:51.228+0000 7f4557104700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:21:51.237+0000 7f4557104700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, reweighted osd.1 to 1 (10000)

2023-03-20T07:14:35.824 WARNING:scrape:Ignoring malparsed backtrace:     running.get()
,   File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get
,   File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception

2023-03-20T07:14:35.824 WARNING:scrape:Ignoring malparsed backtrace:     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
,     next(self.gen)

2023-03-20T07:14:35.824 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2023-03-20T07:14:37.500 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:15:08.583 INFO:tasks.ceph:Archiving crash dumps...
, 2023-03-19T21:15:08.585 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-03-19_18:47:10-crimson-rados-wip-matanb-crimson-only-testing-19.3-v2-distro-crimson-smithi/7213768/remote/smithi149/crash
, 2023-03-19T21:15:12.589 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar cz -f - -C /var/lib/ceph/crash -- .

2023-03-20T07:14:37.500 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:16:26.837 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump
, 2023-03-19T21:16:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1
, 2023-03-19T21:16:27.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive

2023-03-20T07:14:37.684 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f2ed681c7ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2ed681ca2a]

2023-03-20T07:14:38.344 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:14:33.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~31s
, 2023-03-19T20:14:33.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s
, 2023-03-19T20:14:33.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s

2023-03-20T07:14:38.345 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:15:52.552 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
, 2023-03-19T20:15:53.648+0000 7fcfc5247700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:15:53.654+0000 7fcfc5247700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:38.346 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:17:08.925+0000 7f01b47f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:17:08.932+0000 7f01b47f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:17:10.230 INFO:teuthology.orchestra.run.smithi131.stdout:

2023-03-20T07:14:42.581 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:18:31.730+0000 7fdf63d71700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.131:6805/105532 for 29sec e90 v90)
, 2023-03-19T20:18:33.529+0000 7f7be3328700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:18:33.536+0000 7f7be3328700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:43.087 WARNING:scrape:Ignoring malparsed backtrace:     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 455, in run
,     r.wait()

2023-03-20T07:14:43.087 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph.py", line 1463, in run_daemon
,     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/contextutil.py", line 33, in nested

2023-03-20T07:14:43.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:11:30.511 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1
, 2023-03-19T21:11:30.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi132.front.sepia.ceph.com
, 2023-03-19T21:11:31.854 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2

2023-03-20T07:14:43.087 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:12:44.018 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com
, 2023-03-19T21:12:44.018 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
, 2023-03-19T21:12:44.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com

2023-03-20T07:14:47.894 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 18.0.0-3082-g11a4b73e (11a4b73e1cdc89d5296cd7aa42f807e580a3bb41) reef (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f2ed681c7ff]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f2ed681ca2a]

2023-03-20T07:14:49.095 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:14:33.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~31s
, 2023-03-19T20:14:33.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s
, 2023-03-19T20:14:33.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~11s

2023-03-20T07:14:49.096 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:15:52.552 DEBUG:teuthology.orchestra.run.smithi131:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json
, 2023-03-19T20:15:53.648+0000 7fcfc5247700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:15:53.654+0000 7fcfc5247700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:49.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:17:08.925+0000 7f01b47f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:17:08.932+0000 7f01b47f6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:17:10.230 INFO:teuthology.orchestra.run.smithi131.stdout:

2023-03-20T07:14:49.097 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T20:18:31.730+0000 7fdf63d71700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.131:6805/105532 for 29sec e90 v90)
, 2023-03-19T20:18:33.529+0000 7f7be3328700 -1 WARNING: the following dangerous and experimental features are enabled: crimson
, 2023-03-19T20:18:33.536+0000 7f7be3328700 -1 WARNING: the following dangerous and experimental features are enabled: crimson

2023-03-20T07:14:50.364 WARNING:scrape:Ignoring malparsed backtrace:     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 455, in run
,     r.wait()

2023-03-20T07:14:51.532 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_9267f641fd44548732f2529e1d9b8b9ab10a8580/qa/tasks/ceph.py", line 1463, in run_daemon
,     yield
,   File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/contextutil.py", line 33, in nested

2023-03-20T07:14:51.532 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:11:30.511 DEBUG:teuthology.orchestra.run.smithi131:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1
, 2023-03-19T21:11:30.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi132.front.sepia.ceph.com
, 2023-03-19T21:11:31.854 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2

2023-03-20T07:14:51.533 WARNING:scrape:Ignoring malparsed backtrace: 2023-03-19T21:12:44.018 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com
, 2023-03-19T21:12:44.018 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
, 2023-03-19T21:12:44.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com

2023-03-20T07:14:51.695 INFO:scrape:Found 7 distinct failure reasons
2023-03-20T07:14:52.595 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9267f641fd44548732f2529e1d9b8b9ab10a8580 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
2023-03-20T07:14:52.595 INFO:scrape:1 jobs: ['7213769']
2023-03-20T07:14:52.595 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}']
2023-03-20T07:14:52.595 INFO:scrape:
2023-03-20T07:14:52.595 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh
2023-03-20T07:14:52.595 INFO:scrape:1 jobs: ['7213764']
2023-03-20T07:14:52.595 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}']
2023-03-20T07:14:52.595 INFO:scrape:
2023-03-20T07:14:52.595 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2023-03-20T07:14:52.596 INFO:scrape:3 jobs: ['7213770', '7213768', '7213766']
2023-03-20T07:14:52.596 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
2023-03-20T07:14:52.596 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}']
2023-03-20T07:14:52.596 INFO:scrape:
2023-03-20T07:14:52.596 INFO:scrape:Failure: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9267f641fd44548732f2529e1d9b8b9ab10a8580 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh'
2023-03-20T07:14:53.206 INFO:scrape:1 jobs: ['7213755']
2023-03-20T07:14:53.206 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}']
2023-03-20T07:14:53.206 INFO:scrape:
2023-03-20T07:14:53.206 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test_pool_quota.sh
2023-03-20T07:14:53.206 INFO:scrape:1 jobs: ['7213754']
2023-03-20T07:14:53.206 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}']
2023-03-20T07:14:53.206 INFO:scrape:
2023-03-20T07:14:53.206 INFO:scrape:Dead: b'2023-03-20T07:11:11.509 DEBUG:teuthology.exit:Finished running handlers'
2023-03-20T07:14:53.206 INFO:scrape:1 jobs: ['7213757']
2023-03-20T07:14:53.206 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}']
2023-03-20T07:14:53.206 INFO:scrape:
2023-03-20T07:14:53.207 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_lock.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9267f641fd44548732f2529e1d9b8b9ab10a8580 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh'
2023-03-20T07:14:53.207 INFO:scrape:1 jobs: ['7213759']
2023-03-20T07:14:53.207 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}']
2023-03-20T07:14:53.207 INFO:scrape: