2022-09-18T08:29:39.485 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-18T08:29:39.487 INFO:teuthology.results:seed: '6540' 2022-09-18T08:41:26.626 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-18T08:41:26.656 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-18T20:42:24.526 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-09-18T20:42:26.500 INFO:scrape:Found 20 jobs 2022-09-18T20:42:26.613 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f9071f625db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9071f62806] 2022-09-18T20:42:26.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:44.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2022-09-18T09:01:44.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 , 2022-09-18T09:01:44.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-09-18T20:42:26.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:52.612 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-09-18T08:59:49.082765+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:26.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:20.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-09-18T09:02:20.932 INFO:tasks.ceph:Compressing logs... , 2022-09-18T09:02:20.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-09-18T20:42:26.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:24.497 INFO:teuthology.task.internal:Transferring archived files... , 2022-09-18T09:02:24.498 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037234/remote/smithi178 , 2022-09-18T09:02:24.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-09-18T20:42:26.651 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f9071f625db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9071f62806] 2022-09-18T20:42:26.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:44.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2022-09-18T09:01:44.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 , 2022-09-18T09:01:44.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-09-18T20:42:26.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:52.612 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-09-18T08:59:49.082765+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:26.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:20.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-09-18T09:02:20.932 INFO:tasks.ceph:Compressing logs... , 2022-09-18T09:02:20.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-09-18T20:42:26.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:24.497 INFO:teuthology.task.internal:Transferring archived files... , 2022-09-18T09:02:24.498 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037234/remote/smithi178 , 2022-09-18T09:02:24.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-09-18T20:42:26.671 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f9071f625db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f9071f62806] 2022-09-18T20:42:26.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:44.355 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2022-09-18T09:01:44.399 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 , 2022-09-18T09:01:44.491 DEBUG:teuthology.orchestra.run.smithi178:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-09-18T20:42:26.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:01:52.612 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-09-18T08:59:49.082765+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:01:52.613 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2022-09-18T09:00:01.908236+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=0, tm_sec=1, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=1, tm_sec=49, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:26.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:20.892 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-09-18T09:02:20.932 INFO:tasks.ceph:Compressing logs... , 2022-09-18T09:02:20.933 DEBUG:teuthology.orchestra.run.smithi178:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-09-18T20:42:26.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:02:24.497 INFO:teuthology.task.internal:Transferring archived files... , 2022-09-18T09:02:24.498 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037234/remote/smithi178 , 2022-09-18T09:02:24.499 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-09-18T20:42:26.844 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1242712cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55870f23904b] 2022-09-18T20:42:26.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s , 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s , 2022-09-18T08:59:49.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2022-09-18T20:42:26.846 WARNING:scrape:Ignoring malparsed backtrace: return self.get_osd_dump_json()['osds'] , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json , out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2022-09-18T20:42:26.847 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2022-09-18T20:42:26.847 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 139, in stop , run.wait([self.proc], timeout=timeout) 2022-09-18T20:42:26.848 WARNING:scrape:Ignoring malparsed backtrace: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run , r.wait() 2022-09-18T20:42:26.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:10.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2022-09-18T09:04:29.664 INFO:teuthology.task.internal:Transferring binaries for coredumps... , 2022-09-18T09:04:29.675 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037229/remote/smithi005/coredump/1663491547.103626.core.gz: gzip compressed data, was "1663491547.103626.core", last modified: Sun Sep 18 09:04:11 2022, max compression 2022-09-18T20:42:26.869 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1242712cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55870f23904b] 2022-09-18T20:42:26.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s , 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s , 2022-09-18T08:59:49.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2022-09-18T20:42:26.871 WARNING:scrape:Ignoring malparsed backtrace: return self.get_osd_dump_json()['osds'] , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json , out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2022-09-18T20:42:26.872 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2022-09-18T20:42:26.873 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 139, in stop , run.wait([self.proc], timeout=timeout) 2022-09-18T20:42:26.873 WARNING:scrape:Ignoring malparsed backtrace: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run , r.wait() 2022-09-18T20:42:26.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:10.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2022-09-18T09:04:29.664 INFO:teuthology.task.internal:Transferring binaries for coredumps... , 2022-09-18T09:04:29.675 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037229/remote/smithi005/coredump/1663491547.103626.core.gz: gzip compressed data, was "1663491547.103626.core", last modified: Sun Sep 18 09:04:11 2022, max compression 2022-09-18T20:42:26.893 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1242712cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55870f23904b] 2022-09-18T20:42:26.894 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s , 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s , 2022-09-18T08:59:49.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2022-09-18T20:42:26.895 WARNING:scrape:Ignoring malparsed backtrace: return self.get_osd_dump_json()['osds'] , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json , out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2022-09-18T20:42:26.895 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2022-09-18T20:42:26.896 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 139, in stop , run.wait([self.proc], timeout=timeout) 2022-09-18T20:42:26.896 WARNING:scrape:Ignoring malparsed backtrace: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run , r.wait() 2022-09-18T20:42:26.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:10.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2022-09-18T09:04:29.664 INFO:teuthology.task.internal:Transferring binaries for coredumps... , 2022-09-18T09:04:29.675 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037229/remote/smithi005/coredump/1663491547.103626.core.gz: gzip compressed data, was "1663491547.103626.core", last modified: Sun Sep 18 09:04:11 2022, max compression 2022-09-18T20:42:26.925 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f1242712cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x55870f23904b] 2022-09-18T20:42:26.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s , 2022-09-18T08:59:45.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~65s , 2022-09-18T08:59:49.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2022-09-18T20:42:26.928 WARNING:scrape:Ignoring malparsed backtrace: return self.get_osd_dump_json()['osds'] , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 2536, in get_osd_dump_json , out = self.raw_cluster_cmd('osd', 'dump', '--format=json') 2022-09-18T20:42:26.929 WARNING:scrape:Ignoring malparsed backtrace: , During handling of the above exception, another exception occurred: , 2022-09-18T20:42:26.930 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/daemon/state.py", line 139, in stop , run.wait([self.proc], timeout=timeout) 2022-09-18T20:42:26.931 WARNING:scrape:Ignoring malparsed backtrace: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 455, in run , r.wait() 2022-09-18T20:42:26.932 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:10.954 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . , 2022-09-18T09:04:29.664 INFO:teuthology.task.internal:Transferring binaries for coredumps... , 2022-09-18T09:04:29.675 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037229/remote/smithi005/coredump/1663491547.103626.core.gz: gzip compressed data, was "1663491547.103626.core", last modified: Sun Sep 18 09:04:11 2022, max compression 2022-09-18T20:42:27.059 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f68dc6dacb0] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x31f) [0x5579e4dcf4b1] 2022-09-18T20:42:27.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:54:58.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738501 got 34359738495 for osd.2 , 2022-09-18T08:54:58.330 INFO:teuthology.orchestra.run.smithi187.stdout:38654705790 , 2022-09-18T08:54:58.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705796 got 38654705790 for osd.3 2022-09-18T20:42:27.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:55:06.276 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2022-09-18T08:52:55.964097+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=55, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=55, tm_sec=2, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:55:06.276 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-09-18T08:52:48.852513+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=48, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=55, tm_sec=2, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:55:06.277 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-09-18T08:52:44.833700+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=44, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=55, tm_sec=2, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:27.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:55:30.163 INFO:tasks.ceph:Archiving logs... , 2022-09-18T08:55:30.164 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037224/remote/smithi187/log , 2022-09-18T08:55:30.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/log/ceph -- . 2022-09-18T20:42:27.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:55:46.626 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037224/remote/smithi187/coredump/1663491282.102952.core.gz: gzip compressed data, was "1663491282.102952.core", last modified: Sun Sep 18 08:55:32 2022, max compression , , 2022-09-18T08:55:46.627 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2022-09-18T20:42:27.213 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f5593d15cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55983a1533aa] 2022-09-18T20:42:27.214 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-18T08:59:43.197 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3'] , 2022-09-18T08:59:46.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2022-09-18T20:42:27.215 WARNING:scrape:Ignoring malparsed backtrace: out = self.raw_cluster_cmd('osd', 'dump', '--format=json') , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd , return self.run_cluster_cmd(**kwargs).stdout.getvalue() 2022-09-18T20:42:27.216 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph.py", line 1441, in run_daemon 2022-09-18T20:42:27.216 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-09-18T20:42:27.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:07.067 INFO:teuthology.task.internal:Duration was 1140.730258 seconds , 2022-09-18T09:04:07.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-09-18T09:04:07.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-09-18T20:42:27.217 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:28.871 INFO:teuthology.task.internal:Tidying up after the test... , 2022-09-18T09:04:28.872 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2022-09-18T09:04:28.875 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-09-18T20:42:27.236 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f5593d15cb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x588) [0x55983a1533aa] 2022-09-18T20:42:27.237 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:59:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 22 , 2022-09-18T08:59:43.197 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-1, ['config', 'set', 'filestore_inject_stall', '3'] , 2022-09-18T08:59:46.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~77s 2022-09-18T20:42:27.238 WARNING:scrape:Ignoring malparsed backtrace: out = self.raw_cluster_cmd('osd', 'dump', '--format=json') , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd , return self.run_cluster_cmd(**kwargs).stdout.getvalue() 2022-09-18T20:42:27.238 WARNING:scrape:Ignoring malparsed backtrace: , Traceback (most recent call last): , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph.py", line 1441, in run_daemon 2022-09-18T20:42:27.239 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-09-18T20:42:27.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:07.067 INFO:teuthology.task.internal:Duration was 1140.730258 seconds , 2022-09-18T09:04:07.070 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-09-18T09:04:07.088 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-09-18T20:42:27.240 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:28.871 INFO:teuthology.task.internal:Tidying up after the test... , 2022-09-18T09:04:28.872 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest , 2022-09-18T09:04:28.875 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-09-18T20:42:27.498 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f4577d965db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f4577d96806] 2022-09-18T20:42:27.501 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:57:25.803 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2022-09-18T08:57:25.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-09-18T08:57:25.814 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-09-18T20:42:27.502 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:57:33.616 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-09-18T08:52:38.545768+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=38, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:57:33.617 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2022-09-18T08:52:45.576059+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=45, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:57:33.617 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2022-09-18T08:52:47.584384+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=47, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:27.503 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:58:12.740 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037220/remote/smithi096/log , 2022-09-18T08:58:12.742 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/log/ceph -- . , 2022-09-18T08:58:13.827 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037220/remote/smithi104/log 2022-09-18T20:42:27.504 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:58:33.758 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... , 2022-09-18T08:58:35.722 INFO:teuthology.task.internal: core looks like: /tmp/tmpqxhd6e29: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados --pool-snaps --max-ops 4000 --objects 500 --max-in-flight 16 --', real uid: 1000, effective uid: 1000, real gid: 1268, effective gid: 1268, execfn: '/usr/bin/ceph_test_rados', platform: 'x86_64' , 2022-09-18T20:42:27.700 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f1595b7a5db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1595b7a806] 2022-09-18T20:42:27.703 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:38654706000 , 2022-09-18T09:04:44.513 INFO:teuthology.orchestra.run.smithi093.stdout:38654706000 , 2022-09-18T09:04:44.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706005 got 38654706000 for osd.3 2022-09-18T20:42:27.705 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:53.393 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:04:53.394 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:04:53.394 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:27.706 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:05:27.489 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2022-09-18T09:05:27.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2022-09-18T09:05:27.563 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-09-18T20:42:27.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:05:41.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-09-18T09:05:41.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2022-09-18T09:05:41.463 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-09-18T20:42:27.739 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f1595b7a5db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f1595b7a806] 2022-09-18T20:42:27.741 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:44.497 INFO:teuthology.orchestra.run.smithi093.stdout:38654706000 , 2022-09-18T09:04:44.513 INFO:teuthology.orchestra.run.smithi093.stdout:38654706000 , 2022-09-18T09:04:44.516 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654706005 got 38654706000 for osd.3 2022-09-18T20:42:27.742 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:53.393 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:04:53.394 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T09:04:53.394 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2022-09-18T08:59:18.904341+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=59, tm_sec=18, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=9, tm_min=4, tm_sec=50, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:27.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:05:27.489 DEBUG:teuthology.orchestra.run.smithi093:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2022-09-18T09:05:27.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid , 2022-09-18T09:05:27.563 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-09-18T20:42:27.743 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:05:41.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-09-18T09:05:41.446 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2022-09-18T09:05:41.463 DEBUG:teuthology.orchestra.run.smithi093:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-09-18T20:42:27.921 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7f8bc449e5db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8bc449e806] 2022-09-18T20:42:27.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:57:25.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705966 got 38654705960 for osd.3 , 2022-09-18T08:57:25.993 INFO:teuthology.orchestra.run.smithi121.stdout:34359738665 , 2022-09-18T08:57:26.007 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738671 got 34359738665 for osd.0 2022-09-18T20:42:27.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:57:33.652 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-09-18T08:52:26.573262+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=26, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:57:33.652 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2022-09-18T08:52:41.382495+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=41, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) , 2022-09-18T08:57:33.652 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-09-18T08:52:26.573262+0000 time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=52, tm_sec=26, tm_wday=6, tm_yday=261, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=9, tm_mday=18, tm_hour=8, tm_min=57, tm_sec=30, tm_wday=6, tm_yday=261, tm_isdst=0) 2022-09-18T20:42:27.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:58:21.173 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/matan-2022-09-18_08:28:46-crimson-rados:thrash-main-distro-crimson-smithi/7037222/remote/smithi203/log , 2022-09-18T08:58:21.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/log/ceph -- . , 2022-09-18T08:58:21.960 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-09-18T20:42:27.926 WARNING:scrape:Ignoring malparsed backtrace: , 2022-09-18T08:58:41.681 INFO:teuthology.task.internal: dump_program: ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 , 2022-09-18T08:58:41.682 DEBUG:teuthology.orchestra.run.smithi121:> which 'ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000' 2022-09-18T20:42:28.149 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x120) [0x7fdd1ed6d5db] , 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7fdd1ed6d806] 2022-09-18T20:42:28.151 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:56:57.450 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 , 2022-09-18T08:56:57.464 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-09-18T08:56:57.553 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-09-18T20:42:28.152 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T08:57:25.006 INFO:teuthology.orchestra.run.smithi036.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2022-09-18T08:57:24.602991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1081063729,"num_objects":1189,"num_object_clones":687,"num_object_copies":3567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":139,"num_read":890,"num_read_kb":29,"num_write":4560,"num_write_kb":1292541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4435,"ondisk_log_size":4435,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":4025492,"kb_used_data":3905332,"kb_used_omap":0,"kb_used_meta":120128,"kb_avail":373461868,"statfs":{"total":386547056640,"available":382424952832,"internally_reserved":0,"allocated":3999059968,"data_stored":4569201822,"data_compressed":3421926,"data_compressed_allocated":4988928,"data_compressed_original":16072704,"omap_allocated":0,"internal_metadata":123011072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":160,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.688901+0000","last_change":"2022-09-18T08:51:49.799449+0000","last_active":"2022-09-18T08:56:58.688901+0000","last_peered":"2022-09-18T08:56:58.688901+0000","last_clean":"2022-09-18T08:56:58.688901+0000","last_became_active":"2022-09-18T08:51:49.792732+0000","last_became_peered":"2022-09-18T08:51:49.792732+0000","last_unstale":"2022-09-18T08:56:58.688901+0000","last_undegraded":"2022-09-18T08:56:58.688901+0000","last_fullsized":"2022-09-18T08:56:58.688901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.767959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.767959+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.767959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"263'1096","reported_seq":1142,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.812338+0000","last_change":"2022-09-18T08:52:04.742047+0000","last_active":"2022-09-18T08:56:58.812338+0000","last_peered":"2022-09-18T08:56:58.812338+0000","last_clean":"2022-09-18T08:56:58.812338+0000","last_became_active":"2022-09-18T08:52:04.737814+0000","last_became_peered":"2022-09-18T08:52:04.737814+0000","last_unstale":"2022-09-18T08:56:58.812338+0000","last_undegraded":"2022-09-18T08:56:58.812338+0000","last_fullsized":"2022-09-18T08:56:58.812338+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'1096","last_scrub_stamp":"2022-09-18T08:57:03.262504+0000","last_deep_scrub":"263'1096","last_deep_scrub_stamp":"2022-09-18T08:57:03.262504+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.262504+0000","objects_scrubbed":0,"log_size":1096,"log_dups_size":0,"ondisk_log_size":1096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247276860,"num_objects":298,"num_object_clones":177,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":38,"num_read":212,"num_read_kb":0,"num_write":1117,"num_write_kb":323600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":235,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.804421+0000","last_change":"2022-09-18T08:51:49.806529+0000","last_active":"2022-09-18T08:56:58.804421+0000","last_peered":"2022-09-18T08:56:58.804421+0000","last_clean":"2022-09-18T08:56:58.804421+0000","last_became_active":"2022-09-18T08:51:49.802271+0000","last_became_peered":"2022-09-18T08:51:49.802271+0000","last_unstale":"2022-09-18T08:56:58.804421+0000","last_undegraded":"2022-09-18T08:56:58.804421+0000","last_fullsized":"2022-09-18T08:56:58.804421+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:02.744913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:02.744913+0000","last_clean_scrub_stamp":"2022-09-18T08:57:02.744913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":157,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.689124+0000","last_change":"2022-09-18T08:52:02.735852+0000","last_active":"2022-09-18T08:56:58.689124+0000","last_peered":"2022-09-18T08:56:58.689124+0000","last_clean":"2022-09-18T08:56:58.689124+0000","last_became_active":"2022-09-18T08:52:02.732457+0000","last_became_peered":"2022-09-18T08:52:02.732457+0000","last_unstale":"2022-09-18T08:56:58.689124+0000","last_undegraded":"2022-09-18T08:56:58.689124+0000","last_fullsized":"2022-09-18T08:56:58.689124+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.767655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.767655+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.767655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":228,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.821023+0000","last_change":"2022-09-18T08:51:49.756881+0000","last_active":"2022-09-18T08:56:58.821023+0000","last_peered":"2022-09-18T08:56:58.821023+0000","last_clean":"2022-09-18T08:56:58.821023+0000","last_became_active":"2022-09-18T08:51:49.753748+0000","last_became_peered":"2022-09-18T08:51:49.753748+0000","last_unstale":"2022-09-18T08:56:58.821023+0000","last_undegraded":"2022-09-18T08:56:58.821023+0000","last_fullsized":"2022-09-18T08:56:58.821023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.266266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.266266+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.266266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":166,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.734945+0000","last_change":"2022-09-18T08:52:02.705442+0000","last_active":"2022-09-18T08:56:58.734945+0000","last_peered":"2022-09-18T08:56:58.734945+0000","last_clean":"2022-09-18T08:56:58.734945+0000","last_became_active":"2022-09-18T08:52:02.702834+0000","last_became_peered":"2022-09-18T08:52:02.702834+0000","last_unstale":"2022-09-18T08:56:58.734945+0000","last_undegraded":"2022-09-18T08:56:58.734945+0000","last_fullsized":"2022-09-18T08:56:58.734945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:04.277678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:04.277678+0000","last_clean_scrub_stamp":"2022-09-18T08:57:04.277678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":228,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.822226+0000","last_change":"2022-09-18T08:51:49.790709+0000","last_active":"2022-09-18T08:56:58.822226+0000","last_peered":"2022-09-18T08:56:58.822226+0000","last_clean":"2022-09-18T08:56:58.822226+0000","last_became_active":"2022-09-18T08:51:49.787600+0000","last_became_peered":"2022-09-18T08:51:49.787600+0000","last_unstale":"2022-09-18T08:56:58.822226+0000","last_undegraded":"2022-09-18T08:56:58.822226+0000","last_fullsized":"2022-09-18T08:56:58.822226+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.265137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.265137+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.265137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":268,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.739356+0000","last_change":"2022-09-18T08:51:47.750955+0000","last_active":"2022-09-18T08:56:58.739356+0000","last_peered":"2022-09-18T08:56:58.739356+0000","last_clean":"2022-09-18T08:56:58.739356+0000","last_became_active":"2022-09-18T08:51:47.747453+0000","last_became_peered":"2022-09-18T08:51:47.747453+0000","last_unstale":"2022-09-18T08:56:58.739356+0000","last_undegraded":"2022-09-18T08:56:58.739356+0000","last_fullsized":"2022-09-18T08:56:58.739356+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'97","last_scrub_stamp":"2022-09-18T08:57:04.276617+0000","last_deep_scrub":"13'97","last_deep_scrub_stamp":"2022-09-18T08:57:04.276617+0000","last_clean_scrub_stamp":"2022-09-18T08:57:04.276617+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":225,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.821866+0000","last_change":"2022-09-18T08:52:02.709944+0000","last_active":"2022-09-18T08:56:58.821866+0000","last_peered":"2022-09-18T08:56:58.821866+0000","last_clean":"2022-09-18T08:56:58.821866+0000","last_became_active":"2022-09-18T08:52:02.705719+0000","last_became_peered":"2022-09-18T08:52:02.705719+0000","last_unstale":"2022-09-18T08:56:58.821866+0000","last_undegraded":"2022-09-18T08:56:58.821866+0000","last_fullsized":"2022-09-18T08:56:58.821866+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.264058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.264058+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.264058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":235,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.801123+0000","last_change":"2022-09-18T08:51:49.818061+0000","last_active":"2022-09-18T08:56:58.801123+0000","last_peered":"2022-09-18T08:56:58.801123+0000","last_clean":"2022-09-18T08:56:58.801123+0000","last_became_active":"2022-09-18T08:51:49.814900+0000","last_became_peered":"2022-09-18T08:51:49.814900+0000","last_unstale":"2022-09-18T08:56:58.801123+0000","last_undegraded":"2022-09-18T08:56:58.801123+0000","last_fullsized":"2022-09-18T08:56:58.801123+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:02.746001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:02.746001+0000","last_clean_scrub_stamp":"2022-09-18T08:57:02.746001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":166,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.739048+0000","last_change":"2022-09-18T08:52:02.727514+0000","last_active":"2022-09-18T08:56:58.739048+0000","last_peered":"2022-09-18T08:56:58.739048+0000","last_clean":"2022-09-18T08:56:58.739048+0000","last_became_active":"2022-09-18T08:52:02.725249+0000","last_became_peered":"2022-09-18T08:52:02.725249+0000","last_unstale":"2022-09-18T08:56:58.739048+0000","last_undegraded":"2022-09-18T08:56:58.739048+0000","last_fullsized":"2022-09-18T08:56:58.739048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:04.277424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:04.277424+0000","last_clean_scrub_stamp":"2022-09-18T08:57:04.277424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"263'990","reported_seq":1070,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.804073+0000","last_change":"2022-09-18T08:52:04.748166+0000","last_active":"2022-09-18T08:56:58.804073+0000","last_peered":"2022-09-18T08:56:58.804073+0000","last_clean":"2022-09-18T08:56:58.804073+0000","last_became_active":"2022-09-18T08:52:04.745281+0000","last_became_peered":"2022-09-18T08:52:04.745281+0000","last_unstale":"2022-09-18T08:56:58.804073+0000","last_undegraded":"2022-09-18T08:56:58.804073+0000","last_fullsized":"2022-09-18T08:56:58.804073+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'990","last_scrub_stamp":"2022-09-18T08:57:02.745238+0000","last_deep_scrub":"263'990","last_deep_scrub_stamp":"2022-09-18T08:57:02.745238+0000","last_clean_scrub_stamp":"2022-09-18T08:57:02.745238+0000","objects_scrubbed":0,"log_size":990,"log_dups_size":0,"ondisk_log_size":990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268267912,"num_objects":275,"num_object_clones":150,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":37,"num_read":196,"num_read_kb":0,"num_write":1007,"num_write_kb":293162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":169,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.736359+0000","last_change":"2022-09-18T08:51:49.790922+0000","last_active":"2022-09-18T08:56:58.736359+0000","last_peered":"2022-09-18T08:56:58.736359+0000","last_clean":"2022-09-18T08:56:58.736359+0000","last_became_active":"2022-09-18T08:51:49.783842+0000","last_became_peered":"2022-09-18T08:51:49.783842+0000","last_unstale":"2022-09-18T08:56:58.736359+0000","last_undegraded":"2022-09-18T08:56:58.736359+0000","last_fullsized":"2022-09-18T08:56:58.736359+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:04.276995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:04.276995+0000","last_clean_scrub_stamp":"2022-09-18T08:57:04.276995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":160,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.685007+0000","last_change":"2022-09-18T08:51:49.816929+0000","last_active":"2022-09-18T08:56:58.685007+0000","last_peered":"2022-09-18T08:56:58.685007+0000","last_clean":"2022-09-18T08:56:58.685007+0000","last_became_active":"2022-09-18T08:51:49.812423+0000","last_became_peered":"2022-09-18T08:51:49.812423+0000","last_unstale":"2022-09-18T08:56:58.685007+0000","last_undegraded":"2022-09-18T08:56:58.685007+0000","last_fullsized":"2022-09-18T08:56:58.685007+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.767179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.767179+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.767179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"263'1308","reported_seq":1326,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.813042+0000","last_change":"2022-09-18T08:52:04.728322+0000","last_active":"2022-09-18T08:56:58.813042+0000","last_peered":"2022-09-18T08:56:58.813042+0000","last_clean":"2022-09-18T08:56:58.813042+0000","last_became_active":"2022-09-18T08:52:04.725151+0000","last_became_peered":"2022-09-18T08:52:04.725151+0000","last_unstale":"2022-09-18T08:56:58.813042+0000","last_undegraded":"2022-09-18T08:56:58.813042+0000","last_fullsized":"2022-09-18T08:56:58.813042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'1308","last_scrub_stamp":"2022-09-18T08:57:02.746221+0000","last_deep_scrub":"263'1308","last_deep_scrub_stamp":"2022-09-18T08:57:02.746221+0000","last_clean_scrub_stamp":"2022-09-18T08:57:02.746221+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326307961,"num_objects":354,"num_object_clones":212,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":256,"num_read_kb":0,"num_write":1336,"num_write_kb":390917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":228,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.818375+0000","last_change":"2022-09-18T08:51:49.782176+0000","last_active":"2022-09-18T08:56:58.818375+0000","last_peered":"2022-09-18T08:56:58.818375+0000","last_clean":"2022-09-18T08:56:58.818375+0000","last_became_active":"2022-09-18T08:51:49.777716+0000","last_became_peered":"2022-09-18T08:51:49.777716+0000","last_unstale":"2022-09-18T08:56:58.818375+0000","last_undegraded":"2022-09-18T08:56:58.818375+0000","last_fullsized":"2022-09-18T08:56:58.818375+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-09-18T08:57:03.264992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-09-18T08:57:03.264992+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.264992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"259'944","reported_seq":1019,"reported_epoch":272,"state":"active+clean","last_fresh":"2022-09-18T08:56:58.807945+0000","last_change":"2022-09-18T08:52:04.758577+0000","last_active":"2022-09-18T08:56:58.807945+0000","last_peered":"2022-09-18T08:56:58.807945+0000","last_clean":"2022-09-18T08:56:58.807945+0000","last_became_active":"2022-09-18T08:52:04.750678+0000","last_became_peered":"2022-09-18T08:52:04.750678+0000","last_unstale":"2022-09-18T08:56:58.807945+0000","last_undegraded":"2022-09-18T08:56:58.807945+0000","last_fullsized":"2022-09-18T08:56:58.807945+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'944","last_scrub_stamp":"2022-09-18T08:57:03.266047+0000","last_deep_scrub":"259'944","last_deep_scrub_stamp":"2022-09-18T08:57:03.266047+0000","last_clean_scrub_stamp":"2022-09-18T08:57:03.266047+0000","objects_scrubbed":0,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238620628,"num_objects":260,"num_object_clones":148,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":32,"num_read":187,"num_read_kb":0,"num_write":963,"num_write_kb":282968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1080473361,"num_objects":1187,"num_object_clones":687,"num_object_copies":3561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":139,"num_read":851,"num_read_kb":0,"num_write":4423,"num_write_kb":1290647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4338,"ondisk_log_size":4338,"up":12,"acting":12,"num_store_stats":0},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":39,"num_read_kb":29,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654706000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1036824,"kb_used_data":1005776,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":93335016,"statfs":{"total":96636764160,"available":95575056384,"internally_reserved":0,"allocated":1029914624,"data_stored":1181424487,"data_compressed":862959,"data_compressed_allocated":1257472,"data_compressed_original":4161536,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654706000,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":645448,"kb_used_data":623424,"kb_used_omap":0,"kb_used_meta":22016,"kb_avail":93726392,"statfs":{"total":96636764160,"available":95975825408,"internally_reserved":0,"allocated":638386176,"data_stored":733590678,"data_compressed":862959,"data_compressed_allocated":1257472,"data_compressed_original":4161536,"omap_allocated":0,"internal_metadata":22544384},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1004736,"kb_used_data":975224,"kb_used_omap":0,"kb_used_meta":29504,"kb_avail":93367104,"statfs":{"total":96636764160,"available":95607914496,"internally_reserved":0,"allocated":998629376,"data_stored":1133141558,"data_compressed":862959,"data_compressed_allocated":1257472,"data_compressed_original":4161536,"omap_allocated":0,"internal_metadata":30212096},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738705,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":1338484,"kb_used_data":1300908,"kb_used_omap":0,"kb_used_meta":37568,"kb_avail":93033356,"statfs":{"total":96636764160,"available":95266156544,"internally_reserved":0,"allocated":1332129792,"data_stored":1521045099,"data_compressed":833049,"data_compressed_allocated":1216512,"data_compressed_original":3588096,"omap_allocated":0,"internal_metadata":38469632},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} , 2022-09-18T08:57:25.008 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster ceph config set global mon_health_to_clog false , 2022-09-18T08:57:25.357 ERROR:teuthology.contextutil:Saw exception from nested tasks 2022-09-18T20:42:28.153 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/run_tasks.py", line 188, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2022-09-18T20:42:28.154 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: no 16 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) , 2022-09-18T08:58:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 3 , 2022-09-18T08:58:08.251 INFO:teuthology.orchestra.run.smithi036.stdout:/usr/bin/ceph_test_rados 2022-09-18T20:42:28.381 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-14859-ge85e1217 (e85e121747baf9691b263effef4c284a96431457) quincy (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f2de695fcb0] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x229) [0x56013ee0204b] 2022-09-18T20:42:28.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:00:44.223 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 190, in wrapper , return func(self) 2022-09-18T20:42:28.383 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/orchestra/run.py", line 183, in _raise_for_status 2022-09-18T20:42:28.384 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 1609, in raw_cluster_cmd , return self.run_cluster_cmd(**kwargs).stdout.getvalue() , File "/home/teuthworker/src/github.com_Matan-B_ceph_5794c15eeccaa7cf5183e0b7cfbffa4686bab864/qa/tasks/ceph_manager.py", line 1600, in run_cluster_cmd 2022-09-18T20:42:28.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-09-18T09:04:53.868 INFO:teuthology.misc:Shutting down mgr daemons... , 2022-09-18T09:04:53.869 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2022-09-18T09:04:53.870 INFO:teuthology.orchestra.run:waiting for 300 2022-09-18T20:42:28.385 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/contextutil.py", line 33, in nested , yield vars , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/teuthology/task/install/__init__.py", line 619, in task 2022-09-18T20:42:28.385 WARNING:scrape:Ignoring malparsed backtrace: /usr/bin/which: unrecognized option '--objects' , /usr/bin/which: unrecognized option '--max-in-flight' , /usr/bin/which: no 4000 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2022-09-18T20:42:28.393 INFO:scrape:Found 5 distinct failure reasons 2022-09-18T20:42:28.394 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 0 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2022-09-18T20:42:28.395 INFO:scrape:6 jobs: ['7037234', '7037224', '7037220', '7037222', '7037219', '7037233'] 2022-09-18T20:42:28.395 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2022-09-18T20:42:28.396 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/cache-pool-snaps-readproxy}', 'workloads/cache-pool-snaps}', 'workloads/cache-snaps}', 'workloads/pool-snaps-few-objects}', 'workloads/snaps-few-objects-localized}', 'workloads/snaps-few-objects}'] 2022-09-18T20:42:28.396 INFO:scrape: 2022-09-18T20:42:28.397 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2022-09-18T20:42:28.397 INFO:scrape:3 jobs: ['7037229', '7037230', '7037231'] 2022-09-18T20:42:28.398 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2022-09-18T20:42:28.398 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}', 'workloads/small-objects-localized}', 'workloads/small-objects}'] 2022-09-18T20:42:28.399 INFO:scrape: 2022-09-18T20:42:28.399 INFO:scrape:Dead: b'2022-09-18T20:38:50.616 DEBUG:teuthology.exit:Finished running handlers' 2022-09-18T20:42:28.400 INFO:scrape:1 jobs: ['7037221'] 2022-09-18T20:42:28.400 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/cache-snaps-balanced}'] 2022-09-18T20:42:28.401 INFO:scrape: 2022-09-18T20:42:28.401 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2022-09-18T20:42:28.402 INFO:scrape:1 jobs: ['7037225'] 2022-09-18T20:42:28.402 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/rados_api_tests}'] 2022-09-18T20:42:28.403 INFO:scrape: 2022-09-18T20:42:28.404 INFO:scrape:Dead: b"2022-09-18T20:40:49.665 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true" 2022-09-18T20:42:28.404 INFO:scrape:1 jobs: ['7037232'] 2022-09-18T20:42:28.405 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados:thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-balanced}'] 2022-09-18T20:42:28.405 INFO:scrape: