2022-09-18T09:02:34.066 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-18T09:02:34.068 INFO:teuthology.results:seed: '9942'
2022-09-18T09:18:42.647 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18
2022-09-18T09:18:42.675 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-09-18T21:19:39.826 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2022-09-18T21:19:41.497 INFO:scrape:Found 20 jobs
2022-09-18T21:19:41.600 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f386d8e5cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x557462be43aa]

2022-09-18T21:19:41.601 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1422, in _do_thrash
,     self.choose_action()()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1260, in <lambda>

2022-09-18T21:19:41.602 WARNING:scrape:Ignoring malparsed backtrace:     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'
, 2022-09-18T09:38:25.481 DEBUG:teuthology.run_tasks:Unwinding manager ceph

2022-09-18T21:19:41.603 WARNING:scrape:Ignoring malparsed backtrace:     return self.controller.run(**kwargs)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run
,     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)

2022-09-18T21:19:41.604 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'

2022-09-18T21:19:41.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:40:28.623 INFO:teuthology.task.internal.syslog:Compressing syslogs...
, 2022-09-18T09:40:28.624 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-09-18T09:40:28.643 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --

2022-09-18T21:19:41.618 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f386d8e5cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x557462be43aa]

2022-09-18T21:19:41.619 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1422, in _do_thrash
,     self.choose_action()()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1260, in <lambda>

2022-09-18T21:19:41.619 WARNING:scrape:Ignoring malparsed backtrace:     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'
, 2022-09-18T09:38:25.481 DEBUG:teuthology.run_tasks:Unwinding manager ceph

2022-09-18T21:19:41.620 WARNING:scrape:Ignoring malparsed backtrace:     return self.controller.run(**kwargs)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run
,     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)

2022-09-18T21:19:41.621 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'

2022-09-18T21:19:41.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:40:28.623 INFO:teuthology.task.internal.syslog:Compressing syslogs...
, 2022-09-18T09:40:28.624 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-09-18T09:40:28.643 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --

2022-09-18T21:19:41.637 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f386d8e5cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x557462be43aa]

2022-09-18T21:19:41.638 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1422, in _do_thrash
,     self.choose_action()()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 1260, in <lambda>

2022-09-18T21:19:41.639 WARNING:scrape:Ignoring malparsed backtrace:     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi094 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'
, 2022-09-18T09:38:25.481 DEBUG:teuthology.run_tasks:Unwinding manager ceph

2022-09-18T21:19:41.640 WARNING:scrape:Ignoring malparsed backtrace:     return self.controller.run(**kwargs)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run
,     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)

2022-09-18T21:19:41.641 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 179, in _raise_for_status
,     raise CommandCrashedError(command=self.command)
, teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'

2022-09-18T21:19:41.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:40:28.623 INFO:teuthology.task.internal.syslog:Compressing syslogs...
, 2022-09-18T09:40:28.624 DEBUG:teuthology.orchestra.run.smithi094:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --
, 2022-09-18T09:40:28.643 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --

2022-09-18T21:19:41.866 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc3659abcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55cbd67b43aa]

2022-09-18T21:19:41.868 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-18T09:33:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:33:03.100 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['config', 'set', 'filestore_inject_stall', '3']

2022-09-18T21:19:41.868 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-09-18T21:19:41.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:26.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit
, Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1898, in task

2022-09-18T21:19:41.870 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1902, in task

2022-09-18T21:19:41.870 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log'
, 2022-09-18T09:38:26.803 INFO:tasks.ceph.osd.3:Stopped
, 2022-09-18T09:38:26.805 INFO:teuthology.misc:Shutting down mgr daemons...

2022-09-18T21:19:41.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:32.000 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested

2022-09-18T21:19:41.872 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--balance-reads'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

2022-09-18T21:19:41.895 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc3659abcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55cbd67b43aa]

2022-09-18T21:19:41.896 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-18T09:33:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:33:03.100 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['config', 'set', 'filestore_inject_stall', '3']

2022-09-18T21:19:41.897 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-09-18T21:19:41.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:26.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit
, Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1898, in task

2022-09-18T21:19:41.898 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1902, in task

2022-09-18T21:19:41.899 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log'
, 2022-09-18T09:38:26.803 INFO:tasks.ceph.osd.3:Stopped
, 2022-09-18T09:38:26.805 INFO:teuthology.misc:Shutting down mgr daemons...

2022-09-18T21:19:41.899 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:32.000 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested

2022-09-18T21:19:41.900 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--balance-reads'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

2022-09-18T21:19:41.919 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc3659abcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55cbd67b43aa]

2022-09-18T21:19:41.920 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-18T09:33:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:33:03.100 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['config', 'set', 'filestore_inject_stall', '3']

2022-09-18T21:19:41.921 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-09-18T21:19:41.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:26.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit
, Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1898, in task

2022-09-18T21:19:41.922 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1902, in task

2022-09-18T21:19:41.922 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log'
, 2022-09-18T09:38:26.803 INFO:tasks.ceph.osd.3:Stopped
, 2022-09-18T09:38:26.805 INFO:teuthology.misc:Shutting down mgr daemons...

2022-09-18T21:19:41.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:32.000 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested

2022-09-18T21:19:41.924 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--balance-reads'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

2022-09-18T21:19:41.948 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fc3659abcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55cbd67b43aa]

2022-09-18T21:19:41.949 WARNING:scrape:Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-18T09:33:03.099 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:33:03.100 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['config', 'set', 'filestore_inject_stall', '3']

2022-09-18T21:19:41.950 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise
,     raise value.with_traceback(tb)
,   File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run

2022-09-18T21:19:41.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:26.695 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit
, Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1898, in task

2022-09-18T21:19:41.951 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested
,     yield vars
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph.py", line 1902, in task

2022-09-18T21:19:41.952 WARNING:scrape:Ignoring malparsed backtrace: teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log'
, 2022-09-18T09:38:26.803 INFO:tasks.ceph.osd.3:Stopped
, 2022-09-18T09:38:26.805 INFO:teuthology.misc:Shutting down mgr daemons...

2022-09-18T21:19:41.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:38:32.000 ERROR:teuthology.contextutil:Saw exception from nested tasks
, Traceback (most recent call last):
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested

2022-09-18T21:19:41.953 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--balance-reads'
, /usr/bin/which: unrecognized option '--max-ops'
, /usr/bin/which: unrecognized option '--objects'

2022-09-18T21:19:42.155 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7ff737a4fcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55727737104b]

2022-09-18T21:19:42.156 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:36:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:36:18.036 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3']
, 2022-09-18T09:36:21.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s

2022-09-18T21:19:42.157 WARNING:scrape:Ignoring malparsed backtrace:     self.gen.throw(type, value, traceback)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/thrashosds.py", line 215, in task
,     cluster_manager.wait_for_all_osds_up()

2022-09-18T21:19:42.158 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2022-09-18T21:19:42.159 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status

2022-09-18T21:19:42.160 WARNING:scrape:Ignoring malparsed backtrace:     osd_dump = self.get_osd_dump_json()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json
,     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')

2022-09-18T21:19:42.161 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:40:47.199 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core
, 2022-09-18T09:40:47.212 DEBUG:teuthology.orchestra.run.smithi074:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-18T09:40:47.245 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi074.front.sepia.ceph.com, flagging run as failed

2022-09-18T21:19:42.185 WARNING:scrape:Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7ff737a4fcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55727737104b]

2022-09-18T21:19:42.187 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:36:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-18T09:36:18.036 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3']
, 2022-09-18T09:36:21.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~83s

2022-09-18T21:19:42.187 WARNING:scrape:Ignoring malparsed backtrace:     self.gen.throw(type, value, traceback)
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/thrashosds.py", line 215, in task
,     cluster_manager.wait_for_all_osds_up()

2022-09-18T21:19:42.188 WARNING:scrape:Ignoring malparsed backtrace:     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

2022-09-18T21:19:42.189 WARNING:scrape:Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status

2022-09-18T21:19:42.190 WARNING:scrape:Ignoring malparsed backtrace:     osd_dump = self.get_osd_dump_json()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_b17e96d6e4fa0686fff51c2aed8ceb8044d73b3e/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json
,     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')

2022-09-18T21:19:42.190 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:40:47.199 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core
, 2022-09-18T09:40:47.212 DEBUG:teuthology.orchestra.run.smithi074:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-18T09:40:47.245 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi074.front.sepia.ceph.com, flagging run as failed

2022-09-18T21:19:42.552 INFO:scrape:Found 5 distinct failure reasons
2022-09-18T21:19:42.553 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2022-09-18T21:19:42.553 INFO:scrape:1 jobs: ['7037255']
2022-09-18T21:19:42.554 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}']
2022-09-18T21:19:42.554 INFO:scrape:
2022-09-18T21:19:42.555 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2022-09-18T21:19:42.555 INFO:scrape:2 jobs: ['7037251', '7037252']
2022-09-18T21:19:42.556 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
2022-09-18T21:19:42.556 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}']
2022-09-18T21:19:42.557 INFO:scrape:
2022-09-18T21:19:42.557 INFO:scrape:Dead: b"2022-09-18T21:18:20.655 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true"
2022-09-18T21:19:42.557 INFO:scrape:1 jobs: ['7037254']
2022-09-18T21:19:42.558 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}']
2022-09-18T21:19:42.558 INFO:scrape:
2022-09-18T21:19:42.559 INFO:scrape:Dead: b'2022-09-18T21:18:53.168 DEBUG:teuthology.exit:Finished running handlers'
2022-09-18T21:19:42.559 INFO:scrape:1 jobs: ['7037243']
2022-09-18T21:19:42.560 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps-balanced}']
2022-09-18T21:19:42.560 INFO:scrape:
2022-09-18T21:19:42.560 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh
2022-09-18T21:19:42.561 INFO:scrape:1 jobs: ['7037247']
2022-09-18T21:19:42.561 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}']
2022-09-18T21:19:42.562 INFO:scrape: