Found 20 jobs
Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fb07c4a9cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x5601458cf04b]

Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-11T11:51:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-11T11:51:44.619 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3']

Ignoring malparsed backtrace:     cluster_manager.wait_for_all_osds_up()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2791, in wait_for_all_osds_up
,     while not self.are_all_osds_up():

Ignoring malparsed backtrace:     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
,     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

Ignoring malparsed backtrace: 2022-09-11T11:56:06.732 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi005.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T11:56:06.733 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T11:56:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fb07c4a9cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x5601458cf04b]

Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-11T11:51:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-11T11:51:44.619 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3']

Ignoring malparsed backtrace:     cluster_manager.wait_for_all_osds_up()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2791, in wait_for_all_osds_up
,     while not self.are_all_osds_up():

Ignoring malparsed backtrace:     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
,     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

Ignoring malparsed backtrace: 2022-09-11T11:56:06.732 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi005.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T11:56:06.733 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T11:56:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fb07c4a9cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x5601458cf04b]

Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-11T11:51:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-11T11:51:44.619 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3']

Ignoring malparsed backtrace:     cluster_manager.wait_for_all_osds_up()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2791, in wait_for_all_osds_up
,     while not self.are_all_osds_up():

Ignoring malparsed backtrace:     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
,     node=self.hostname, label=self.label
, teuthology.exceptions.CommandFailedError: Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

Ignoring malparsed backtrace:     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd
,     return self.run_cluster_cmd(**kwargs).stdout.getvalue()

Ignoring malparsed backtrace: 2022-09-11T11:56:06.732 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi005.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T11:56:06.733 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T11:56:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f91e7bbdcb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x56100944104b]

Ignoring malparsed backtrace: admin_socket: exception getting command descriptions: [Errno 111] Connection refused
, 2022-09-11T11:51:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 22
, 2022-09-11T11:51:05.198 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['config', 'set', 'filestore_inject_stall', '3']

Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2544, in get_osd_dump
,     return self.get_osd_dump_json()['osds']
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json

Ignoring malparsed backtrace: 
, Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph.py", line 1441, in run_daemon

Ignoring malparsed backtrace:     run.wait([self.proc], timeout=timeout)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/remote.py", line 510, in run
,     r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run

Ignoring malparsed backtrace: 2022-09-11T11:55:21.799 DEBUG:teuthology.misc:Transferring archived files from smithi093:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2022-09-11_11:21:46-crimson-rados:thrash-main-distro-crimson-smithi/7026140/remote/smithi093
, 2022-09-11T11:55:21.800 DEBUG:teuthology.orchestra.run.smithi093:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
, 2022-09-11T11:55:35.449 INFO:teuthology.task.internal:Transferring binaries for coredumps...

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52f7a325db]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52f7a32806]

Ignoring malparsed backtrace: 2022-09-11T11:59:16.446 INFO:teuthology.orchestra.run.smithi088.stdout:34359738915
, 2022-09-11T11:59:16.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738916 got 34359738915 for osd.0
, 2022-09-11T11:59:17.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1

Ignoring malparsed backtrace: 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.298 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-09-11T11:50:11.912525+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=11, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)

Ignoring malparsed backtrace: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory
, 2022-09-11T11:59:59.559 INFO:tasks.ceph:Archiving crash dumps...
, 2022-09-11T11:59:59.567 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-09-11_11:21:46-crimson-rados:thrash-main-distro-crimson-smithi/7026139/remote/smithi088/crash

Ignoring malparsed backtrace: 2022-09-11T12:00:12.154 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T12:00:12.194 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi088.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T12:00:12.196 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52f7a325db]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52f7a32806]

Ignoring malparsed backtrace: 2022-09-11T11:59:16.446 INFO:teuthology.orchestra.run.smithi088.stdout:34359738915
, 2022-09-11T11:59:16.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738916 got 34359738915 for osd.0
, 2022-09-11T11:59:17.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1

Ignoring malparsed backtrace: 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.298 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-09-11T11:50:11.912525+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=11, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)

Ignoring malparsed backtrace: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory
, 2022-09-11T11:59:59.559 INFO:tasks.ceph:Archiving crash dumps...
, 2022-09-11T11:59:59.567 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-09-11_11:21:46-crimson-rados:thrash-main-distro-crimson-smithi/7026139/remote/smithi088/crash

Ignoring malparsed backtrace: 2022-09-11T12:00:12.154 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T12:00:12.194 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi088.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T12:00:12.196 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52f7a325db]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52f7a32806]

Ignoring malparsed backtrace: 2022-09-11T11:59:16.446 INFO:teuthology.orchestra.run.smithi088.stdout:34359738915
, 2022-09-11T11:59:16.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738916 got 34359738915 for osd.0
, 2022-09-11T11:59:17.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1

Ignoring malparsed backtrace: 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.298 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-09-11T11:50:11.912525+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=11, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)

Ignoring malparsed backtrace: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory
, 2022-09-11T11:59:59.559 INFO:tasks.ceph:Archiving crash dumps...
, 2022-09-11T11:59:59.567 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-09-11_11:21:46-crimson-rados:thrash-main-distro-crimson-smithi/7026139/remote/smithi088/crash

Ignoring malparsed backtrace: 2022-09-11T12:00:12.154 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T12:00:12.194 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi088.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T12:00:12.196 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f52f7a325db]
,  2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f52f7a32806]

Ignoring malparsed backtrace: 2022-09-11T11:59:16.446 INFO:teuthology.orchestra.run.smithi088.stdout:34359738915
, 2022-09-11T11:59:16.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738916 got 34359738915 for osd.0
, 2022-09-11T11:59:17.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1

Ignoring malparsed backtrace: 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.297 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2022-09-11T11:50:18.939860+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=18, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)
, 2022-09-11T11:59:25.298 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-09-11T11:50:11.912525+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=50, tm_sec=11, tm_wday=6, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=11, tm_hour=11, tm_min=59, tm_sec=21, tm_wday=6, tm_yday=254, tm_isdst=0)

Ignoring malparsed backtrace: gzip: /var/log/ceph/valgrind/*.gz: No such file or directory
, 2022-09-11T11:59:59.559 INFO:tasks.ceph:Archiving crash dumps...
, 2022-09-11T11:59:59.567 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2022-09-11_11:21:46-crimson-rados:thrash-main-distro-crimson-smithi/7026139/remote/smithi088/crash

Ignoring malparsed backtrace: 2022-09-11T12:00:12.154 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump
, 2022-09-11T12:00:12.194 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi088.front.sepia.ceph.com, flagging run as failed
, 2022-09-11T12:00:12.196 DEBUG:teuthology.orchestra.run.smithi090:> test -e /home/ubuntu/cephtest/archive/coredump

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7fb8e8136cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x560e3fb5604b]

Ignoring malparsed backtrace: 2022-09-11T11:41:38.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~65s
, 2022-09-11T11:41:38.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s
, 2022-09-11T11:41:39.180 DEBUG:teuthology.orchestra.run.smithi139:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3

Ignoring malparsed backtrace:     suppress = manager.__exit__(*exc_info)
,   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
,     self.gen.throw(type, value, traceback)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 479, in wait
,     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait

Ignoring malparsed backtrace:     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status
,     node=self.hostname, label=self.label

Ignoring malparsed backtrace: 2022-09-11T11:45:58.564 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2
, 2022-09-11T11:45:58.659 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi139.front.sepia.ceph.com
, 2022-09-11T11:45:58.661 DEBUG:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3

Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/rados.py", line 274, in thread
,     run.wait(tests.values())
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 479, in wait

Ignoring malparsed backtrace:   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/rados.py", line 286, in task
,     running.get()
,   File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f694a9c2cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x56508df8f04b]

Ignoring malparsed backtrace: 2022-09-11T11:50:42.204 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 190, in wrapper
,     return func(self)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status

Ignoring malparsed backtrace:     osd_dump = self.get_osd_dump_json()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json
,     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')

Ignoring malparsed backtrace:     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()

Ignoring malparsed backtrace: 2022-09-11T11:54:51.725 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
, 2022-09-11T11:54:51.751 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com
, 2022-09-11T11:54:51.752 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

Ignoring malparsed backtrace:  ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev)
,  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xcc) [0x7f694a9c2cb0]
,  2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x56508df8f04b]

Ignoring malparsed backtrace: 2022-09-11T11:50:42.204 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 190, in wrapper
,     return func(self)

Ignoring malparsed backtrace:   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status

Ignoring malparsed backtrace:     osd_dump = self.get_osd_dump_json()
,   File "/home/teuthworker/src/github.com_Matan-B_ceph_0b4e0fe792f883d7efc0ac426337711af142b78e/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json
,     out = self.raw_cluster_cmd('osd', 'dump', '--format=json')

Ignoring malparsed backtrace:     proc.wait()
,   File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait
,     self._raise_for_status()

Ignoring malparsed backtrace: 2022-09-11T11:54:51.725 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1
, 2022-09-11T11:54:51.751 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com
, 2022-09-11T11:54:51.752 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1

Found 4 distinct failure reasons
Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
3 jobs: ['7026141', '7026140', '7026129']
suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects-localized}']

Dead: b'2022-09-11T23:31:19.145 DEBUG:teuthology.exit:Finished running handlers'
5 jobs: ['7026127', '7026128', '7026132', '7026133', '7026130']
suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_api_tests}']

Dead: b"2022-09-11T23:36:16.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true"
2 jobs: ['7026137', '7026142']
suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects}']

Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
2 jobs: ['7026139', '7026138']
suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health']
suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}', 'workloads/small-objects}']