Found 20 jobs Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7efe7362ced4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5585824970ca] Ignoring malparsed backtrace: 2023-06-09T02:46:33.414+0000 7f5390f22700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , set osd.3 primary-affinity to 1 (8655362) , 2023-06-09T02:46:34.266 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet Ignoring malparsed backtrace: 2023-06-09T02:47:32.230+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.236+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.587 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:48:33.122+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.128+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.342 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:49:33.890+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:33.897+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:34.115 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:50:34.511+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.518+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.733 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd , return self.controller.run(**kwargs) , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/remote.py", line 525, in run Ignoring malparsed backtrace: 2023-06-09T03:43:07.792 INFO:tasks.ceph.osd.1:Stopped , 2023-06-09T03:43:07.792 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2023-06-09T03:43:07.793 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.125875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129215.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.125875.log.gz , /var/log/ceph/ceph-client.admin.111947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125119.log Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.107961.log.gz , /var/log/ceph/ceph-client.admin.112221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117391.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.112221.log.gz Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.119234.log.gz , /var/log/ceph/ceph-client.admin.113468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112760.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.113468.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.109425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127385.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.109425.log.gz , /var/log/ceph/ceph-client.admin.108669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111510.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112921.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124845.log.gz , /var/log/ceph/ceph-client.admin.116852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112865.log Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107145.log.gz , /var/log/ceph/ceph-client.admin.123815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121105.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.123815.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126808.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.115718.log.gz , /var/log/ceph/ceph-client.admin.113524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125336.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.130007.log: gzip -5 --verbose -- /var/log/ceph/ceph.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.130007.log.gz , /var/log/ceph/ceph-client.admin.121049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130350.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124620.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124620.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122758.log , /var/log/ceph/ceph-client.admin.109677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115549.log Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.118477.log.gz , /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116635.log , /var/log/ceph/ceph-client.admin.129607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119802.log Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 274, in thread , run.wait(tests.values()) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 286, in task , running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7efe7362ced4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5585824970ca] Ignoring malparsed backtrace: 2023-06-09T02:46:33.414+0000 7f5390f22700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , set osd.3 primary-affinity to 1 (8655362) , 2023-06-09T02:46:34.266 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet Ignoring malparsed backtrace: 2023-06-09T02:47:32.230+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.236+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.587 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:48:33.122+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.128+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.342 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:49:33.890+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:33.897+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:34.115 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:50:34.511+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.518+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.733 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd , return self.controller.run(**kwargs) , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/remote.py", line 525, in run Ignoring malparsed backtrace: 2023-06-09T03:43:07.792 INFO:tasks.ceph.osd.1:Stopped , 2023-06-09T03:43:07.792 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2023-06-09T03:43:07.793 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.125875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129215.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.125875.log.gz , /var/log/ceph/ceph-client.admin.111947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125119.log Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.107961.log.gz , /var/log/ceph/ceph-client.admin.112221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117391.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.112221.log.gz Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.119234.log.gz , /var/log/ceph/ceph-client.admin.113468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112760.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.113468.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.109425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127385.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.109425.log.gz , /var/log/ceph/ceph-client.admin.108669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111510.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112921.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124845.log.gz , /var/log/ceph/ceph-client.admin.116852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112865.log Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107145.log.gz , /var/log/ceph/ceph-client.admin.123815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121105.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.123815.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126808.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.115718.log.gz , /var/log/ceph/ceph-client.admin.113524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125336.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.130007.log: gzip -5 --verbose -- /var/log/ceph/ceph.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.130007.log.gz , /var/log/ceph/ceph-client.admin.121049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130350.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124620.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124620.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122758.log , /var/log/ceph/ceph-client.admin.109677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115549.log Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.118477.log.gz , /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116635.log , /var/log/ceph/ceph-client.admin.129607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119802.log Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 274, in thread , run.wait(tests.values()) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 286, in task , running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7efe7362ced4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x5585824970ca] Ignoring malparsed backtrace: 2023-06-09T02:46:33.414+0000 7f5390f22700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , set osd.3 primary-affinity to 1 (8655362) , 2023-06-09T02:46:34.266 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet Ignoring malparsed backtrace: 2023-06-09T02:47:32.230+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.236+0000 7f7291ede700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:32.587 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:48:33.122+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.128+0000 7f3b7cf6a700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:48:33.342 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:49:33.890+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:33.897+0000 7fd712bbd700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:34.115 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: 2023-06-09T02:50:34.511+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.518+0000 7feaef858700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:50:34.733 INFO:teuthology.orchestra.run.smithi092.stdout: Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/ceph_manager.py", line 1588, in run_cluster_cmd , return self.controller.run(**kwargs) , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/teuthology/orchestra/remote.py", line 525, in run Ignoring malparsed backtrace: 2023-06-09T03:43:07.792 INFO:tasks.ceph.osd.1:Stopped , 2023-06-09T03:43:07.792 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2023-06-09T03:43:07.793 INFO:teuthology.orchestra.run:waiting for 300 Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.125875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129215.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.125875.log.gz , /var/log/ceph/ceph-client.admin.111947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125119.log Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.107961.log.gz , /var/log/ceph/ceph-client.admin.112221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117391.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.112221.log.gz Ignoring malparsed backtrace: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.119234.log.gz , /var/log/ceph/ceph-client.admin.113468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112760.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.113468.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.109425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127385.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.109425.log.gz , /var/log/ceph/ceph-client.admin.108669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111510.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112921.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124845.log.gz , /var/log/ceph/ceph-client.admin.116852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112865.log Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107145.log.gz , /var/log/ceph/ceph-client.admin.123815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121105.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.123815.log.gz Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.115718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126808.log , 50.4% -- replaced with /var/log/ceph/ceph-client.admin.115718.log.gz , /var/log/ceph/ceph-client.admin.113524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125336.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.130007.log: gzip -5 --verbose -- /var/log/ceph/ceph.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.130007.log.gz , /var/log/ceph/ceph-client.admin.121049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.130350.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.124620.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.124620.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122758.log , /var/log/ceph/ceph-client.admin.109677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115549.log Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.118477.log.gz , /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116635.log , /var/log/ceph/ceph-client.admin.129607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119802.log Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 274, in thread , run.wait(tests.values()) Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_ceph-c_65a3a7bbcb9f2381b9a62987589a7b08fdf694d1/qa/tasks/rados.py", line 286, in task , running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f45890a4ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x563addee0535] Ignoring malparsed backtrace: 2023-06-09T02:46:37.766 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'stale+active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:46:33.652594+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:37.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2023-06-09T02:46:37.886+0000 7f5736193700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:46:46.664 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:PG 3.8 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:46:51.858 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:46:51.872 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-06-09T02:46:51.761104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":22,"state":"active+undersized+wait","last_fresh":"2023-06-09T02:46:34.700340+0000","last_change":"2023-06-09T02:46:34.699896+0000","last_active":"2023-06-09T02:46:34.700340+0000","last_peered":"2023-06-09T02:46:34.700340+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.700340+0000","last_undegraded":"2023-06-09T02:46:34.700340+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.707422+0000', 'last_change': '2023-06-09T02:46:47.725692+0000', 'last_active': '2023-06-09T02:46:34.707422+0000', 'last_peered': '2023-06-09T02:46:34.707422+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:46:34.699896+0000', 'last_became_peered': '2023-06-09T02:46:34.699896+0000', 'last_unstale': '2023-06-09T02:46:34.707422+0000', 'last_undegraded': '2023-06-09T02:46:34.707422+0000', 'last_fullsized': '2023-06-09T02:46:34.691990+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:08.705+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.712+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.907 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean , 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "20'291", 'reported_seq': 787, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.710414+0000', 'last_change': '2023-06-09T02:46:41.715854+0000', 'last_active': '2023-06-09T02:46:34.710414+0000', 'last_peered': '2023-06-09T02:46:34.710414+0000', 'last_clean': '2023-06-09T02:46:33.654852+0000', 'last_became_active': '2023-06-09T02:46:34.696619+0000', 'last_became_peered': '2023-06-09T02:46:34.696619+0000', 'last_unstale': '2023-06-09T02:46:34.710414+0000', 'last_undegraded': '2023-06-09T02:46:34.689238+0000', 'last_fullsized': '2023-06-09T02:46:34.688652+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 291, 'log_dups_size': 0, 'ondisk_log_size': 291, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 143776516, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 48, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 460, 'num_read_kb': 249496, 'num_write': 700, 'num_write_kb': 201710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 48}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:13.384 INFO:tasks.ceph.ceph_manager.ceph:PG 3.7 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:22.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "20'212", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.703770+0000', 'last_change': '2023-06-09T02:46:41.720448+0000', 'last_active': '2023-06-09T02:46:34.703770+0000', 'last_peered': '2023-06-09T02:46:34.703770+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.696111+0000', 'last_became_peered': '2023-06-09T02:46:34.696111+0000', 'last_unstale': '2023-06-09T02:46:34.703770+0000', 'last_undegraded': '2023-06-09T02:46:34.703770+0000', 'last_fullsized': '2023-06-09T02:46:34.687473+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 212, 'log_dups_size': 0, 'ondisk_log_size': 212, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "21'272", 'reported_seq': 205, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692992+0000', 'last_change': '2023-06-09T02:46:25.404450+0000', 'last_active': '2023-06-09T02:46:34.692992+0000', 'last_peered': '2023-06-09T02:46:34.692992+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.718786+0000', 'last_became_peered': '2023-06-09T02:45:52.718786+0000', 'last_unstale': '2023-06-09T02:46:34.692992+0000', 'last_undegraded': '2023-06-09T02:46:25.403808+0000', 'last_fullsized': '2023-06-09T02:45:52.707174+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 272, 'log_dups_size': 0, 'ondisk_log_size': 272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5111308, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 104749, 'num_write': 162, 'num_write_kb': 49828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:26.873+0000 7f4cb1e57700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout: , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c3a25994-a927-4d0a-bff8-7be9672f8e22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"OSD_HOST_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 host (2 osds) down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 318/932 objects degraded (34.120%), 10 pgs degraded, 13 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":22,"num_osds":4,"num_up_osds":2,"osd_up_since":1686278793,"num_in_osds":4,"osd_in_since":1686278645,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":10},{"state_name":"active+undersized","count":10},{"state_name":"active+clean","count":5}],"num_pgs":25,"num_pools":3,"num_objects":466,"data_bytes":1375932054,"bytes_used":1566179328,"bytes_avail":427930550272,"bytes_total":429496729600,"degraded_objects":318,"degraded_total":932,"degraded_ratio":0.34120171673819738},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-06-09T02:45:11.738171+0000","services":{}},"progress_events":{}} Ignoring malparsed backtrace: 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.8', 'version': "20'359", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.709224+0000', 'last_change': '2023-06-09T02:46:41.747822+0000', 'last_active': '2023-06-09T02:46:34.709224+0000', 'last_peered': '2023-06-09T02:46:34.709224+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.702955+0000', 'last_became_peered': '2023-06-09T02:46:34.702955+0000', 'last_unstale': '2023-06-09T02:46:34.709224+0000', 'last_undegraded': '2023-06-09T02:46:34.709224+0000', 'last_fullsized': '2023-06-09T02:46:34.686747+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 359, 'log_dups_size': 0, 'ondisk_log_size': 359, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean , 2023-06-09T02:47:35.638 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "20'302", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.706704+0000', 'last_change': '2023-06-09T02:46:41.773676+0000', 'last_active': '2023-06-09T02:46:34.706704+0000', 'last_peered': '2023-06-09T02:46:34.706704+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.701324+0000', 'last_became_peered': '2023-06-09T02:46:34.701324+0000', 'last_unstale': '2023-06-09T02:46:34.706704+0000', 'last_undegraded': '2023-06-09T02:46:34.706704+0000', 'last_fullsized': '2023-06-09T02:46:34.689653+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 302, 'log_dups_size': 0, 'ondisk_log_size': 302, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:43.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:47:43.988+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:43.994+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:47:48.995 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.714638+0000', 'last_change': '2023-06-09T02:45:59.729905+0000', 'last_active': '2023-06-09T02:45:52.714638+0000', 'last_peered': '2023-06-09T02:45:52.714638+0000', 'last_clean': '2023-06-09T02:44:27.665796+0000', 'last_became_active': '2023-06-09T02:45:52.708834+0000', 'last_became_peered': '2023-06-09T02:45:52.708834+0000', 'last_unstale': '2023-06-09T02:45:52.714638+0000', 'last_undegraded': '2023-06-09T02:45:52.714638+0000', 'last_fullsized': '2023-06-09T02:45:52.703512+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "13'97", 'reported_seq': 255, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692045+0000', 'last_change': '2023-06-09T02:46:05.710796+0000', 'last_active': '2023-06-09T02:46:34.692045+0000', 'last_peered': '2023-06-09T02:46:34.692045+0000', 'last_clean': '2023-06-09T02:44:27.701670+0000', 'last_became_active': '2023-06-09T02:45:52.716766+0000', 'last_became_peered': '2023-06-09T02:45:52.716766+0000', 'last_unstale': '2023-06-09T02:46:34.692045+0000', 'last_undegraded': '2023-06-09T02:45:52.711660+0000', 'last_fullsized': '2023-06-09T02:45:52.711198+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1180736, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 142, 'num_read_kb': 120, 'num_write': 274, 'num_write_kb': 3788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: dumped all , 2023-06-09T02:47:57.904 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2023-06-09T02:47:57.779026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} , 2023-06-09T02:47:57.906 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:02.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.7', 'version': "20'336", 'reported_seq': 932, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.705105+0000', 'last_change': '2023-06-09T02:46:41.736133+0000', 'last_active': '2023-06-09T02:46:34.705105+0000', 'last_peered': '2023-06-09T02:46:34.705105+0000', 'last_clean': '2023-06-09T02:46:33.628600+0000', 'last_became_active': '2023-06-09T02:46:34.699062+0000', 'last_became_peered': '2023-06-09T02:46:34.699062+0000', 'last_unstale': '2023-06-09T02:46:34.705105+0000', 'last_undegraded': '2023-06-09T02:46:34.692479+0000', 'last_fullsized': '2023-06-09T02:46:34.691837+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 336, 'log_dups_size': 0, 'ondisk_log_size': 336, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 165494196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 583, 'num_read_kb': 325012, 'num_write': 816, 'num_write_kb': 241588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 60}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682233+0000', 'last_change': '2023-06-09T02:45:59.713810+0000', 'last_active': '2023-06-09T02:46:34.682233+0000', 'last_peered': '2023-06-09T02:46:34.682233+0000', 'last_clean': '2023-06-09T02:44:31.719829+0000', 'last_became_active': '2023-06-09T02:45:52.714365+0000', 'last_became_peered': '2023-06-09T02:45:52.714365+0000', 'last_unstale': '2023-06-09T02:46:34.682233+0000', 'last_undegraded': '2023-06-09T02:46:34.682233+0000', 'last_fullsized': '2023-06-09T02:45:52.708114+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.b is not active+clean , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.b', 'version': "21'242", 'reported_seq': 688, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.679503+0000', 'last_change': '2023-06-09T02:46:01.720164+0000', 'last_active': '2023-06-09T02:46:34.679503+0000', 'last_peered': '2023-06-09T02:46:34.679503+0000', 'last_clean': '2023-06-09T02:45:50.161878+0000', 'last_became_active': '2023-06-09T02:45:52.718038+0000', 'last_became_peered': '2023-06-09T02:45:52.718038+0000', 'last_unstale': '2023-06-09T02:46:34.679503+0000', 'last_undegraded': '2023-06-09T02:45:52.707209+0000', 'last_fullsized': '2023-06-09T02:45:52.706167+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 242, 'log_dups_size': 0, 'ondisk_log_size': 242, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 97926294, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 459, 'num_read_kb': 300322, 'num_write': 580, 'num_write_kb': 168476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 34}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:16.120 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now , 2023-06-09T02:48:16.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:48:16.240+0000 7fa1994d9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:48:24.603 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682773+0000', 'last_change': '2023-06-09T02:45:59.757634+0000', 'last_active': '2023-06-09T02:46:34.682773+0000', 'last_peered': '2023-06-09T02:46:34.682773+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:45:52.729380+0000', 'last_became_peered': '2023-06-09T02:45:52.729380+0000', 'last_unstale': '2023-06-09T02:46:34.682773+0000', 'last_undegraded': '2023-06-09T02:46:34.682773+0000', 'last_fullsized': '2023-06-09T02:45:52.718827+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:33.153 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:48:33.166 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2023-06-09T02:48:31.789080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "20'256", 'reported_seq': 748, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.391628+0000', 'last_change': '2023-06-09T02:46:01.725103+0000', 'last_active': '2023-06-09T02:46:34.391628+0000', 'last_peered': '2023-06-09T02:46:34.391628+0000', 'last_clean': '2023-06-09T02:45:50.291566+0000', 'last_became_active': '2023-06-09T02:45:52.705375+0000', 'last_became_peered': '2023-06-09T02:45:52.705375+0000', 'last_unstale': '2023-06-09T02:46:34.391628+0000', 'last_undegraded': '2023-06-09T02:45:52.701032+0000', 'last_fullsized': '2023-06-09T02:45:52.700626+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 256, 'log_dups_size': 0, 'ondisk_log_size': 256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 150971298, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 50, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 535, 'num_read_kb': 343900, 'num_write': 634, 'num_write_kb': 185274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 50}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:46.860 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "20'259", 'reported_seq': 112, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.248083+0000', 'last_change': '2023-06-09T02:46:01.770104+0000', 'last_active': '2023-06-09T02:46:34.248083+0000', 'last_peered': '2023-06-09T02:46:34.248083+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707814+0000', 'last_became_peered': '2023-06-09T02:45:52.707814+0000', 'last_unstale': '2023-06-09T02:46:34.248083+0000', 'last_undegraded': '2023-06-09T02:46:34.248083+0000', 'last_fullsized': '2023-06-09T02:45:52.700919+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 259, 'log_dups_size': 0, 'ondisk_log_size': 259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3260756, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 125, 'num_read_kb': 108716, 'num_write': 62, 'num_write_kb': 18824, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "20'310", 'reported_seq': 844, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.687596+0000', 'last_change': '2023-06-09T02:46:41.724545+0000', 'last_active': '2023-06-09T02:46:34.687596+0000', 'last_peered': '2023-06-09T02:46:34.687596+0000', 'last_clean': '2023-06-09T02:46:32.996280+0000', 'last_became_active': '2023-06-09T02:46:34.684268+0000', 'last_became_peered': '2023-06-09T02:46:34.684268+0000', 'last_unstale': '2023-06-09T02:46:34.687596+0000', 'last_undegraded': '2023-06-09T02:46:34.680280+0000', 'last_fullsized': '2023-06-09T02:46:34.679638+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 310, 'log_dups_size': 0, 'ondisk_log_size': 310, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 185529196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 485, 'num_read_kb': 265896, 'num_write': 770, 'num_write_kb': 225044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:51.326 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean , 2023-06-09T02:48:51.327 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:45:52.717013+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:51.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json Ignoring malparsed backtrace: 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "20'236", 'reported_seq': 151, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:33.474100+0000', 'last_change': '2023-06-09T02:46:10.887913+0000', 'last_active': '2023-06-09T02:46:33.474100+0000', 'last_peered': '2023-06-09T02:46:33.474100+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707238+0000', 'last_became_peered': '2023-06-09T02:45:52.707238+0000', 'last_unstale': '2023-06-09T02:46:33.474100+0000', 'last_undegraded': '2023-06-09T02:46:10.887514+0000', 'last_fullsized': '2023-06-09T02:45:52.700718+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 236, 'log_dups_size': 0, 'ondisk_log_size': 236, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21349840, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 103816, 'num_write': 106, 'num_write_kb': 31122, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:49:00.228 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:05.221+0000 7f4524fc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:05.414 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all Ignoring malparsed backtrace: 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'313", 'reported_seq': 849, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.782700+0000', 'last_change': '2023-06-09T02:49:11.782700+0000', 'last_active': '2023-06-09T02:49:11.782700+0000', 'last_peered': '2023-06-09T02:46:34.681801+0000', 'last_clean': '2023-06-09T02:45:51.547062+0000', 'last_became_active': '2023-06-09T02:45:52.724794+0000', 'last_became_peered': '2023-06-09T02:45:52.724794+0000', 'last_unstale': '2023-06-09T02:49:11.782700+0000', 'last_undegraded': '2023-06-09T02:49:11.782700+0000', 'last_fullsized': '2023-06-09T02:49:11.782700+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 313, 'log_dups_size': 0, 'ondisk_log_size': 313, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 147073196, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 504, 'num_read_kb': 233526, 'num_write': 764, 'num_write_kb': 216182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.813946+0000', 'last_change': '2023-06-09T02:49:11.743688+0000', 'last_active': '2023-06-09T02:49:11.743688+0000', 'last_peered': '2023-06-09T02:45:52.702176+0000', 'last_clean': '2023-06-09T02:45:52.702176+0000', 'last_became_active': '2023-06-09T02:44:15.722764+0000', 'last_became_peered': '2023-06-09T02:44:15.722764+0000', 'last_unstale': '2023-06-09T02:49:11.813946+0000', 'last_undegraded': '2023-06-09T02:49:11.813946+0000', 'last_fullsized': '2023-06-09T02:49:11.813946+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:27.085+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.092+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.287 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.106366.log.gz , /var/log/ceph/ceph-client.admin.107963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110663.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107963.log.gz Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.115335.log.gz , /var/log/ceph/ceph-client.admin.117001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112240.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.117001.log.gz Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109781.log , /var/log/ceph/ceph-client.admin.107596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107596.log.gz , /var/log/ceph/ceph-client.admin.114053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107794.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.108330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113465.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.108330.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116756.log: /var/log/ceph/ceph-client.admin.112191.log Ignoring malparsed backtrace: gzip /var/log/ceph/ceph-client.admin.110173.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.112338.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.110173.log.gz , /var/log/ceph/ceph-client.admin.109340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111056.log Ignoring malparsed backtrace: 2023-06-09T02:50:19.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2023-06-09T02:50:19.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-06-09T02:50:19.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f45890a4ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x563addee0535] Ignoring malparsed backtrace: 2023-06-09T02:46:37.766 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'stale+active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:46:33.652594+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:37.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2023-06-09T02:46:37.886+0000 7f5736193700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:46:46.664 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:PG 3.8 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:46:51.858 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:46:51.872 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-06-09T02:46:51.761104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":22,"state":"active+undersized+wait","last_fresh":"2023-06-09T02:46:34.700340+0000","last_change":"2023-06-09T02:46:34.699896+0000","last_active":"2023-06-09T02:46:34.700340+0000","last_peered":"2023-06-09T02:46:34.700340+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.700340+0000","last_undegraded":"2023-06-09T02:46:34.700340+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.707422+0000', 'last_change': '2023-06-09T02:46:47.725692+0000', 'last_active': '2023-06-09T02:46:34.707422+0000', 'last_peered': '2023-06-09T02:46:34.707422+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:46:34.699896+0000', 'last_became_peered': '2023-06-09T02:46:34.699896+0000', 'last_unstale': '2023-06-09T02:46:34.707422+0000', 'last_undegraded': '2023-06-09T02:46:34.707422+0000', 'last_fullsized': '2023-06-09T02:46:34.691990+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:08.705+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.712+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.907 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean , 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "20'291", 'reported_seq': 787, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.710414+0000', 'last_change': '2023-06-09T02:46:41.715854+0000', 'last_active': '2023-06-09T02:46:34.710414+0000', 'last_peered': '2023-06-09T02:46:34.710414+0000', 'last_clean': '2023-06-09T02:46:33.654852+0000', 'last_became_active': '2023-06-09T02:46:34.696619+0000', 'last_became_peered': '2023-06-09T02:46:34.696619+0000', 'last_unstale': '2023-06-09T02:46:34.710414+0000', 'last_undegraded': '2023-06-09T02:46:34.689238+0000', 'last_fullsized': '2023-06-09T02:46:34.688652+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 291, 'log_dups_size': 0, 'ondisk_log_size': 291, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 143776516, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 48, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 460, 'num_read_kb': 249496, 'num_write': 700, 'num_write_kb': 201710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 48}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:13.384 INFO:tasks.ceph.ceph_manager.ceph:PG 3.7 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:22.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "20'212", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.703770+0000', 'last_change': '2023-06-09T02:46:41.720448+0000', 'last_active': '2023-06-09T02:46:34.703770+0000', 'last_peered': '2023-06-09T02:46:34.703770+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.696111+0000', 'last_became_peered': '2023-06-09T02:46:34.696111+0000', 'last_unstale': '2023-06-09T02:46:34.703770+0000', 'last_undegraded': '2023-06-09T02:46:34.703770+0000', 'last_fullsized': '2023-06-09T02:46:34.687473+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 212, 'log_dups_size': 0, 'ondisk_log_size': 212, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "21'272", 'reported_seq': 205, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692992+0000', 'last_change': '2023-06-09T02:46:25.404450+0000', 'last_active': '2023-06-09T02:46:34.692992+0000', 'last_peered': '2023-06-09T02:46:34.692992+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.718786+0000', 'last_became_peered': '2023-06-09T02:45:52.718786+0000', 'last_unstale': '2023-06-09T02:46:34.692992+0000', 'last_undegraded': '2023-06-09T02:46:25.403808+0000', 'last_fullsized': '2023-06-09T02:45:52.707174+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 272, 'log_dups_size': 0, 'ondisk_log_size': 272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5111308, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 104749, 'num_write': 162, 'num_write_kb': 49828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:26.873+0000 7f4cb1e57700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout: , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c3a25994-a927-4d0a-bff8-7be9672f8e22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"OSD_HOST_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 host (2 osds) down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 318/932 objects degraded (34.120%), 10 pgs degraded, 13 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":22,"num_osds":4,"num_up_osds":2,"osd_up_since":1686278793,"num_in_osds":4,"osd_in_since":1686278645,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":10},{"state_name":"active+undersized","count":10},{"state_name":"active+clean","count":5}],"num_pgs":25,"num_pools":3,"num_objects":466,"data_bytes":1375932054,"bytes_used":1566179328,"bytes_avail":427930550272,"bytes_total":429496729600,"degraded_objects":318,"degraded_total":932,"degraded_ratio":0.34120171673819738},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-06-09T02:45:11.738171+0000","services":{}},"progress_events":{}} Ignoring malparsed backtrace: 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.8', 'version': "20'359", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.709224+0000', 'last_change': '2023-06-09T02:46:41.747822+0000', 'last_active': '2023-06-09T02:46:34.709224+0000', 'last_peered': '2023-06-09T02:46:34.709224+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.702955+0000', 'last_became_peered': '2023-06-09T02:46:34.702955+0000', 'last_unstale': '2023-06-09T02:46:34.709224+0000', 'last_undegraded': '2023-06-09T02:46:34.709224+0000', 'last_fullsized': '2023-06-09T02:46:34.686747+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 359, 'log_dups_size': 0, 'ondisk_log_size': 359, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean , 2023-06-09T02:47:35.638 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "20'302", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.706704+0000', 'last_change': '2023-06-09T02:46:41.773676+0000', 'last_active': '2023-06-09T02:46:34.706704+0000', 'last_peered': '2023-06-09T02:46:34.706704+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.701324+0000', 'last_became_peered': '2023-06-09T02:46:34.701324+0000', 'last_unstale': '2023-06-09T02:46:34.706704+0000', 'last_undegraded': '2023-06-09T02:46:34.706704+0000', 'last_fullsized': '2023-06-09T02:46:34.689653+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 302, 'log_dups_size': 0, 'ondisk_log_size': 302, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:43.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:47:43.988+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:43.994+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:47:48.995 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.714638+0000', 'last_change': '2023-06-09T02:45:59.729905+0000', 'last_active': '2023-06-09T02:45:52.714638+0000', 'last_peered': '2023-06-09T02:45:52.714638+0000', 'last_clean': '2023-06-09T02:44:27.665796+0000', 'last_became_active': '2023-06-09T02:45:52.708834+0000', 'last_became_peered': '2023-06-09T02:45:52.708834+0000', 'last_unstale': '2023-06-09T02:45:52.714638+0000', 'last_undegraded': '2023-06-09T02:45:52.714638+0000', 'last_fullsized': '2023-06-09T02:45:52.703512+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "13'97", 'reported_seq': 255, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692045+0000', 'last_change': '2023-06-09T02:46:05.710796+0000', 'last_active': '2023-06-09T02:46:34.692045+0000', 'last_peered': '2023-06-09T02:46:34.692045+0000', 'last_clean': '2023-06-09T02:44:27.701670+0000', 'last_became_active': '2023-06-09T02:45:52.716766+0000', 'last_became_peered': '2023-06-09T02:45:52.716766+0000', 'last_unstale': '2023-06-09T02:46:34.692045+0000', 'last_undegraded': '2023-06-09T02:45:52.711660+0000', 'last_fullsized': '2023-06-09T02:45:52.711198+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1180736, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 142, 'num_read_kb': 120, 'num_write': 274, 'num_write_kb': 3788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: dumped all , 2023-06-09T02:47:57.904 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2023-06-09T02:47:57.779026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} , 2023-06-09T02:47:57.906 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:02.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.7', 'version': "20'336", 'reported_seq': 932, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.705105+0000', 'last_change': '2023-06-09T02:46:41.736133+0000', 'last_active': '2023-06-09T02:46:34.705105+0000', 'last_peered': '2023-06-09T02:46:34.705105+0000', 'last_clean': '2023-06-09T02:46:33.628600+0000', 'last_became_active': '2023-06-09T02:46:34.699062+0000', 'last_became_peered': '2023-06-09T02:46:34.699062+0000', 'last_unstale': '2023-06-09T02:46:34.705105+0000', 'last_undegraded': '2023-06-09T02:46:34.692479+0000', 'last_fullsized': '2023-06-09T02:46:34.691837+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 336, 'log_dups_size': 0, 'ondisk_log_size': 336, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 165494196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 583, 'num_read_kb': 325012, 'num_write': 816, 'num_write_kb': 241588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 60}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682233+0000', 'last_change': '2023-06-09T02:45:59.713810+0000', 'last_active': '2023-06-09T02:46:34.682233+0000', 'last_peered': '2023-06-09T02:46:34.682233+0000', 'last_clean': '2023-06-09T02:44:31.719829+0000', 'last_became_active': '2023-06-09T02:45:52.714365+0000', 'last_became_peered': '2023-06-09T02:45:52.714365+0000', 'last_unstale': '2023-06-09T02:46:34.682233+0000', 'last_undegraded': '2023-06-09T02:46:34.682233+0000', 'last_fullsized': '2023-06-09T02:45:52.708114+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.b is not active+clean , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.b', 'version': "21'242", 'reported_seq': 688, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.679503+0000', 'last_change': '2023-06-09T02:46:01.720164+0000', 'last_active': '2023-06-09T02:46:34.679503+0000', 'last_peered': '2023-06-09T02:46:34.679503+0000', 'last_clean': '2023-06-09T02:45:50.161878+0000', 'last_became_active': '2023-06-09T02:45:52.718038+0000', 'last_became_peered': '2023-06-09T02:45:52.718038+0000', 'last_unstale': '2023-06-09T02:46:34.679503+0000', 'last_undegraded': '2023-06-09T02:45:52.707209+0000', 'last_fullsized': '2023-06-09T02:45:52.706167+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 242, 'log_dups_size': 0, 'ondisk_log_size': 242, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 97926294, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 459, 'num_read_kb': 300322, 'num_write': 580, 'num_write_kb': 168476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 34}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:16.120 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now , 2023-06-09T02:48:16.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:48:16.240+0000 7fa1994d9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:48:24.603 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682773+0000', 'last_change': '2023-06-09T02:45:59.757634+0000', 'last_active': '2023-06-09T02:46:34.682773+0000', 'last_peered': '2023-06-09T02:46:34.682773+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:45:52.729380+0000', 'last_became_peered': '2023-06-09T02:45:52.729380+0000', 'last_unstale': '2023-06-09T02:46:34.682773+0000', 'last_undegraded': '2023-06-09T02:46:34.682773+0000', 'last_fullsized': '2023-06-09T02:45:52.718827+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:33.153 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:48:33.166 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2023-06-09T02:48:31.789080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "20'256", 'reported_seq': 748, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.391628+0000', 'last_change': '2023-06-09T02:46:01.725103+0000', 'last_active': '2023-06-09T02:46:34.391628+0000', 'last_peered': '2023-06-09T02:46:34.391628+0000', 'last_clean': '2023-06-09T02:45:50.291566+0000', 'last_became_active': '2023-06-09T02:45:52.705375+0000', 'last_became_peered': '2023-06-09T02:45:52.705375+0000', 'last_unstale': '2023-06-09T02:46:34.391628+0000', 'last_undegraded': '2023-06-09T02:45:52.701032+0000', 'last_fullsized': '2023-06-09T02:45:52.700626+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 256, 'log_dups_size': 0, 'ondisk_log_size': 256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 150971298, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 50, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 535, 'num_read_kb': 343900, 'num_write': 634, 'num_write_kb': 185274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 50}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:46.860 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "20'259", 'reported_seq': 112, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.248083+0000', 'last_change': '2023-06-09T02:46:01.770104+0000', 'last_active': '2023-06-09T02:46:34.248083+0000', 'last_peered': '2023-06-09T02:46:34.248083+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707814+0000', 'last_became_peered': '2023-06-09T02:45:52.707814+0000', 'last_unstale': '2023-06-09T02:46:34.248083+0000', 'last_undegraded': '2023-06-09T02:46:34.248083+0000', 'last_fullsized': '2023-06-09T02:45:52.700919+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 259, 'log_dups_size': 0, 'ondisk_log_size': 259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3260756, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 125, 'num_read_kb': 108716, 'num_write': 62, 'num_write_kb': 18824, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "20'310", 'reported_seq': 844, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.687596+0000', 'last_change': '2023-06-09T02:46:41.724545+0000', 'last_active': '2023-06-09T02:46:34.687596+0000', 'last_peered': '2023-06-09T02:46:34.687596+0000', 'last_clean': '2023-06-09T02:46:32.996280+0000', 'last_became_active': '2023-06-09T02:46:34.684268+0000', 'last_became_peered': '2023-06-09T02:46:34.684268+0000', 'last_unstale': '2023-06-09T02:46:34.687596+0000', 'last_undegraded': '2023-06-09T02:46:34.680280+0000', 'last_fullsized': '2023-06-09T02:46:34.679638+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 310, 'log_dups_size': 0, 'ondisk_log_size': 310, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 185529196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 485, 'num_read_kb': 265896, 'num_write': 770, 'num_write_kb': 225044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:51.326 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean , 2023-06-09T02:48:51.327 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:45:52.717013+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:51.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json Ignoring malparsed backtrace: 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "20'236", 'reported_seq': 151, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:33.474100+0000', 'last_change': '2023-06-09T02:46:10.887913+0000', 'last_active': '2023-06-09T02:46:33.474100+0000', 'last_peered': '2023-06-09T02:46:33.474100+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707238+0000', 'last_became_peered': '2023-06-09T02:45:52.707238+0000', 'last_unstale': '2023-06-09T02:46:33.474100+0000', 'last_undegraded': '2023-06-09T02:46:10.887514+0000', 'last_fullsized': '2023-06-09T02:45:52.700718+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 236, 'log_dups_size': 0, 'ondisk_log_size': 236, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21349840, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 103816, 'num_write': 106, 'num_write_kb': 31122, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:49:00.228 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:05.221+0000 7f4524fc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:05.414 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all Ignoring malparsed backtrace: 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'313", 'reported_seq': 849, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.782700+0000', 'last_change': '2023-06-09T02:49:11.782700+0000', 'last_active': '2023-06-09T02:49:11.782700+0000', 'last_peered': '2023-06-09T02:46:34.681801+0000', 'last_clean': '2023-06-09T02:45:51.547062+0000', 'last_became_active': '2023-06-09T02:45:52.724794+0000', 'last_became_peered': '2023-06-09T02:45:52.724794+0000', 'last_unstale': '2023-06-09T02:49:11.782700+0000', 'last_undegraded': '2023-06-09T02:49:11.782700+0000', 'last_fullsized': '2023-06-09T02:49:11.782700+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 313, 'log_dups_size': 0, 'ondisk_log_size': 313, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 147073196, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 504, 'num_read_kb': 233526, 'num_write': 764, 'num_write_kb': 216182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.813946+0000', 'last_change': '2023-06-09T02:49:11.743688+0000', 'last_active': '2023-06-09T02:49:11.743688+0000', 'last_peered': '2023-06-09T02:45:52.702176+0000', 'last_clean': '2023-06-09T02:45:52.702176+0000', 'last_became_active': '2023-06-09T02:44:15.722764+0000', 'last_became_peered': '2023-06-09T02:44:15.722764+0000', 'last_unstale': '2023-06-09T02:49:11.813946+0000', 'last_undegraded': '2023-06-09T02:49:11.813946+0000', 'last_fullsized': '2023-06-09T02:49:11.813946+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:27.085+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.092+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.287 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.106366.log.gz , /var/log/ceph/ceph-client.admin.107963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110663.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107963.log.gz Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.115335.log.gz , /var/log/ceph/ceph-client.admin.117001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112240.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.117001.log.gz Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109781.log , /var/log/ceph/ceph-client.admin.107596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107596.log.gz , /var/log/ceph/ceph-client.admin.114053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107794.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.108330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113465.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.108330.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116756.log: /var/log/ceph/ceph-client.admin.112191.log Ignoring malparsed backtrace: gzip /var/log/ceph/ceph-client.admin.110173.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.112338.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.110173.log.gz , /var/log/ceph/ceph-client.admin.109340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111056.log Ignoring malparsed backtrace: 2023-06-09T02:50:19.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2023-06-09T02:50:19.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-06-09T02:50:19.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f45890a4ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x563addee0535] Ignoring malparsed backtrace: 2023-06-09T02:46:37.766 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'stale+active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:46:33.652594+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:37.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2023-06-09T02:46:37.886+0000 7f5736193700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:46:46.664 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:PG 3.8 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:46:51.858 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:46:51.872 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-06-09T02:46:51.761104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":22,"state":"active+undersized+wait","last_fresh":"2023-06-09T02:46:34.700340+0000","last_change":"2023-06-09T02:46:34.699896+0000","last_active":"2023-06-09T02:46:34.700340+0000","last_peered":"2023-06-09T02:46:34.700340+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.700340+0000","last_undegraded":"2023-06-09T02:46:34.700340+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.707422+0000', 'last_change': '2023-06-09T02:46:47.725692+0000', 'last_active': '2023-06-09T02:46:34.707422+0000', 'last_peered': '2023-06-09T02:46:34.707422+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:46:34.699896+0000', 'last_became_peered': '2023-06-09T02:46:34.699896+0000', 'last_unstale': '2023-06-09T02:46:34.707422+0000', 'last_undegraded': '2023-06-09T02:46:34.707422+0000', 'last_fullsized': '2023-06-09T02:46:34.691990+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:08.705+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.712+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.907 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean , 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "20'291", 'reported_seq': 787, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.710414+0000', 'last_change': '2023-06-09T02:46:41.715854+0000', 'last_active': '2023-06-09T02:46:34.710414+0000', 'last_peered': '2023-06-09T02:46:34.710414+0000', 'last_clean': '2023-06-09T02:46:33.654852+0000', 'last_became_active': '2023-06-09T02:46:34.696619+0000', 'last_became_peered': '2023-06-09T02:46:34.696619+0000', 'last_unstale': '2023-06-09T02:46:34.710414+0000', 'last_undegraded': '2023-06-09T02:46:34.689238+0000', 'last_fullsized': '2023-06-09T02:46:34.688652+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 291, 'log_dups_size': 0, 'ondisk_log_size': 291, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 143776516, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 48, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 460, 'num_read_kb': 249496, 'num_write': 700, 'num_write_kb': 201710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 48}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:13.384 INFO:tasks.ceph.ceph_manager.ceph:PG 3.7 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:22.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "20'212", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.703770+0000', 'last_change': '2023-06-09T02:46:41.720448+0000', 'last_active': '2023-06-09T02:46:34.703770+0000', 'last_peered': '2023-06-09T02:46:34.703770+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.696111+0000', 'last_became_peered': '2023-06-09T02:46:34.696111+0000', 'last_unstale': '2023-06-09T02:46:34.703770+0000', 'last_undegraded': '2023-06-09T02:46:34.703770+0000', 'last_fullsized': '2023-06-09T02:46:34.687473+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 212, 'log_dups_size': 0, 'ondisk_log_size': 212, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "21'272", 'reported_seq': 205, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692992+0000', 'last_change': '2023-06-09T02:46:25.404450+0000', 'last_active': '2023-06-09T02:46:34.692992+0000', 'last_peered': '2023-06-09T02:46:34.692992+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.718786+0000', 'last_became_peered': '2023-06-09T02:45:52.718786+0000', 'last_unstale': '2023-06-09T02:46:34.692992+0000', 'last_undegraded': '2023-06-09T02:46:25.403808+0000', 'last_fullsized': '2023-06-09T02:45:52.707174+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 272, 'log_dups_size': 0, 'ondisk_log_size': 272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5111308, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 104749, 'num_write': 162, 'num_write_kb': 49828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:26.873+0000 7f4cb1e57700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout: , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c3a25994-a927-4d0a-bff8-7be9672f8e22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"OSD_HOST_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 host (2 osds) down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 318/932 objects degraded (34.120%), 10 pgs degraded, 13 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":22,"num_osds":4,"num_up_osds":2,"osd_up_since":1686278793,"num_in_osds":4,"osd_in_since":1686278645,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":10},{"state_name":"active+undersized","count":10},{"state_name":"active+clean","count":5}],"num_pgs":25,"num_pools":3,"num_objects":466,"data_bytes":1375932054,"bytes_used":1566179328,"bytes_avail":427930550272,"bytes_total":429496729600,"degraded_objects":318,"degraded_total":932,"degraded_ratio":0.34120171673819738},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-06-09T02:45:11.738171+0000","services":{}},"progress_events":{}} Ignoring malparsed backtrace: 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.8', 'version': "20'359", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.709224+0000', 'last_change': '2023-06-09T02:46:41.747822+0000', 'last_active': '2023-06-09T02:46:34.709224+0000', 'last_peered': '2023-06-09T02:46:34.709224+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.702955+0000', 'last_became_peered': '2023-06-09T02:46:34.702955+0000', 'last_unstale': '2023-06-09T02:46:34.709224+0000', 'last_undegraded': '2023-06-09T02:46:34.709224+0000', 'last_fullsized': '2023-06-09T02:46:34.686747+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 359, 'log_dups_size': 0, 'ondisk_log_size': 359, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean , 2023-06-09T02:47:35.638 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "20'302", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.706704+0000', 'last_change': '2023-06-09T02:46:41.773676+0000', 'last_active': '2023-06-09T02:46:34.706704+0000', 'last_peered': '2023-06-09T02:46:34.706704+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.701324+0000', 'last_became_peered': '2023-06-09T02:46:34.701324+0000', 'last_unstale': '2023-06-09T02:46:34.706704+0000', 'last_undegraded': '2023-06-09T02:46:34.706704+0000', 'last_fullsized': '2023-06-09T02:46:34.689653+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 302, 'log_dups_size': 0, 'ondisk_log_size': 302, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:43.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:47:43.988+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:43.994+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:47:48.995 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.714638+0000', 'last_change': '2023-06-09T02:45:59.729905+0000', 'last_active': '2023-06-09T02:45:52.714638+0000', 'last_peered': '2023-06-09T02:45:52.714638+0000', 'last_clean': '2023-06-09T02:44:27.665796+0000', 'last_became_active': '2023-06-09T02:45:52.708834+0000', 'last_became_peered': '2023-06-09T02:45:52.708834+0000', 'last_unstale': '2023-06-09T02:45:52.714638+0000', 'last_undegraded': '2023-06-09T02:45:52.714638+0000', 'last_fullsized': '2023-06-09T02:45:52.703512+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "13'97", 'reported_seq': 255, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692045+0000', 'last_change': '2023-06-09T02:46:05.710796+0000', 'last_active': '2023-06-09T02:46:34.692045+0000', 'last_peered': '2023-06-09T02:46:34.692045+0000', 'last_clean': '2023-06-09T02:44:27.701670+0000', 'last_became_active': '2023-06-09T02:45:52.716766+0000', 'last_became_peered': '2023-06-09T02:45:52.716766+0000', 'last_unstale': '2023-06-09T02:46:34.692045+0000', 'last_undegraded': '2023-06-09T02:45:52.711660+0000', 'last_fullsized': '2023-06-09T02:45:52.711198+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1180736, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 142, 'num_read_kb': 120, 'num_write': 274, 'num_write_kb': 3788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: dumped all , 2023-06-09T02:47:57.904 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2023-06-09T02:47:57.779026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} , 2023-06-09T02:47:57.906 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:02.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.7', 'version': "20'336", 'reported_seq': 932, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.705105+0000', 'last_change': '2023-06-09T02:46:41.736133+0000', 'last_active': '2023-06-09T02:46:34.705105+0000', 'last_peered': '2023-06-09T02:46:34.705105+0000', 'last_clean': '2023-06-09T02:46:33.628600+0000', 'last_became_active': '2023-06-09T02:46:34.699062+0000', 'last_became_peered': '2023-06-09T02:46:34.699062+0000', 'last_unstale': '2023-06-09T02:46:34.705105+0000', 'last_undegraded': '2023-06-09T02:46:34.692479+0000', 'last_fullsized': '2023-06-09T02:46:34.691837+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 336, 'log_dups_size': 0, 'ondisk_log_size': 336, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 165494196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 583, 'num_read_kb': 325012, 'num_write': 816, 'num_write_kb': 241588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 60}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682233+0000', 'last_change': '2023-06-09T02:45:59.713810+0000', 'last_active': '2023-06-09T02:46:34.682233+0000', 'last_peered': '2023-06-09T02:46:34.682233+0000', 'last_clean': '2023-06-09T02:44:31.719829+0000', 'last_became_active': '2023-06-09T02:45:52.714365+0000', 'last_became_peered': '2023-06-09T02:45:52.714365+0000', 'last_unstale': '2023-06-09T02:46:34.682233+0000', 'last_undegraded': '2023-06-09T02:46:34.682233+0000', 'last_fullsized': '2023-06-09T02:45:52.708114+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.b is not active+clean , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.b', 'version': "21'242", 'reported_seq': 688, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.679503+0000', 'last_change': '2023-06-09T02:46:01.720164+0000', 'last_active': '2023-06-09T02:46:34.679503+0000', 'last_peered': '2023-06-09T02:46:34.679503+0000', 'last_clean': '2023-06-09T02:45:50.161878+0000', 'last_became_active': '2023-06-09T02:45:52.718038+0000', 'last_became_peered': '2023-06-09T02:45:52.718038+0000', 'last_unstale': '2023-06-09T02:46:34.679503+0000', 'last_undegraded': '2023-06-09T02:45:52.707209+0000', 'last_fullsized': '2023-06-09T02:45:52.706167+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 242, 'log_dups_size': 0, 'ondisk_log_size': 242, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 97926294, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 459, 'num_read_kb': 300322, 'num_write': 580, 'num_write_kb': 168476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 34}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:16.120 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now , 2023-06-09T02:48:16.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:48:16.240+0000 7fa1994d9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:48:24.603 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682773+0000', 'last_change': '2023-06-09T02:45:59.757634+0000', 'last_active': '2023-06-09T02:46:34.682773+0000', 'last_peered': '2023-06-09T02:46:34.682773+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:45:52.729380+0000', 'last_became_peered': '2023-06-09T02:45:52.729380+0000', 'last_unstale': '2023-06-09T02:46:34.682773+0000', 'last_undegraded': '2023-06-09T02:46:34.682773+0000', 'last_fullsized': '2023-06-09T02:45:52.718827+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:33.153 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:48:33.166 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2023-06-09T02:48:31.789080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "20'256", 'reported_seq': 748, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.391628+0000', 'last_change': '2023-06-09T02:46:01.725103+0000', 'last_active': '2023-06-09T02:46:34.391628+0000', 'last_peered': '2023-06-09T02:46:34.391628+0000', 'last_clean': '2023-06-09T02:45:50.291566+0000', 'last_became_active': '2023-06-09T02:45:52.705375+0000', 'last_became_peered': '2023-06-09T02:45:52.705375+0000', 'last_unstale': '2023-06-09T02:46:34.391628+0000', 'last_undegraded': '2023-06-09T02:45:52.701032+0000', 'last_fullsized': '2023-06-09T02:45:52.700626+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 256, 'log_dups_size': 0, 'ondisk_log_size': 256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 150971298, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 50, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 535, 'num_read_kb': 343900, 'num_write': 634, 'num_write_kb': 185274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 50}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:46.860 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "20'259", 'reported_seq': 112, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.248083+0000', 'last_change': '2023-06-09T02:46:01.770104+0000', 'last_active': '2023-06-09T02:46:34.248083+0000', 'last_peered': '2023-06-09T02:46:34.248083+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707814+0000', 'last_became_peered': '2023-06-09T02:45:52.707814+0000', 'last_unstale': '2023-06-09T02:46:34.248083+0000', 'last_undegraded': '2023-06-09T02:46:34.248083+0000', 'last_fullsized': '2023-06-09T02:45:52.700919+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 259, 'log_dups_size': 0, 'ondisk_log_size': 259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3260756, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 125, 'num_read_kb': 108716, 'num_write': 62, 'num_write_kb': 18824, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "20'310", 'reported_seq': 844, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.687596+0000', 'last_change': '2023-06-09T02:46:41.724545+0000', 'last_active': '2023-06-09T02:46:34.687596+0000', 'last_peered': '2023-06-09T02:46:34.687596+0000', 'last_clean': '2023-06-09T02:46:32.996280+0000', 'last_became_active': '2023-06-09T02:46:34.684268+0000', 'last_became_peered': '2023-06-09T02:46:34.684268+0000', 'last_unstale': '2023-06-09T02:46:34.687596+0000', 'last_undegraded': '2023-06-09T02:46:34.680280+0000', 'last_fullsized': '2023-06-09T02:46:34.679638+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 310, 'log_dups_size': 0, 'ondisk_log_size': 310, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 185529196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 485, 'num_read_kb': 265896, 'num_write': 770, 'num_write_kb': 225044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:51.326 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean , 2023-06-09T02:48:51.327 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:45:52.717013+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:51.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json Ignoring malparsed backtrace: 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "20'236", 'reported_seq': 151, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:33.474100+0000', 'last_change': '2023-06-09T02:46:10.887913+0000', 'last_active': '2023-06-09T02:46:33.474100+0000', 'last_peered': '2023-06-09T02:46:33.474100+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707238+0000', 'last_became_peered': '2023-06-09T02:45:52.707238+0000', 'last_unstale': '2023-06-09T02:46:33.474100+0000', 'last_undegraded': '2023-06-09T02:46:10.887514+0000', 'last_fullsized': '2023-06-09T02:45:52.700718+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 236, 'log_dups_size': 0, 'ondisk_log_size': 236, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21349840, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 103816, 'num_write': 106, 'num_write_kb': 31122, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:49:00.228 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:05.221+0000 7f4524fc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:05.414 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all Ignoring malparsed backtrace: 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'313", 'reported_seq': 849, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.782700+0000', 'last_change': '2023-06-09T02:49:11.782700+0000', 'last_active': '2023-06-09T02:49:11.782700+0000', 'last_peered': '2023-06-09T02:46:34.681801+0000', 'last_clean': '2023-06-09T02:45:51.547062+0000', 'last_became_active': '2023-06-09T02:45:52.724794+0000', 'last_became_peered': '2023-06-09T02:45:52.724794+0000', 'last_unstale': '2023-06-09T02:49:11.782700+0000', 'last_undegraded': '2023-06-09T02:49:11.782700+0000', 'last_fullsized': '2023-06-09T02:49:11.782700+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 313, 'log_dups_size': 0, 'ondisk_log_size': 313, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 147073196, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 504, 'num_read_kb': 233526, 'num_write': 764, 'num_write_kb': 216182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.813946+0000', 'last_change': '2023-06-09T02:49:11.743688+0000', 'last_active': '2023-06-09T02:49:11.743688+0000', 'last_peered': '2023-06-09T02:45:52.702176+0000', 'last_clean': '2023-06-09T02:45:52.702176+0000', 'last_became_active': '2023-06-09T02:44:15.722764+0000', 'last_became_peered': '2023-06-09T02:44:15.722764+0000', 'last_unstale': '2023-06-09T02:49:11.813946+0000', 'last_undegraded': '2023-06-09T02:49:11.813946+0000', 'last_fullsized': '2023-06-09T02:49:11.813946+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:27.085+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.092+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.287 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.106366.log.gz , /var/log/ceph/ceph-client.admin.107963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110663.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107963.log.gz Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.115335.log.gz , /var/log/ceph/ceph-client.admin.117001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112240.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.117001.log.gz Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109781.log , /var/log/ceph/ceph-client.admin.107596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107596.log.gz , /var/log/ceph/ceph-client.admin.114053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107794.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.108330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113465.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.108330.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116756.log: /var/log/ceph/ceph-client.admin.112191.log Ignoring malparsed backtrace: gzip /var/log/ceph/ceph-client.admin.110173.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.112338.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.110173.log.gz , /var/log/ceph/ceph-client.admin.109340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111056.log Ignoring malparsed backtrace: 2023-06-09T02:50:19.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2023-06-09T02:50:19.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-06-09T02:50:19.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz Ignoring malparsed backtrace: ceph version 18.0.0-4318-g65a3a7bb (65a3a7bbcb9f2381b9a62987589a7b08fdf694d1) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7f45890a4ed4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x61b) [0x563addee0535] Ignoring malparsed backtrace: 2023-06-09T02:46:37.766 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'stale+active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:46:33.652594+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:37.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2023-06-09T02:46:37.886+0000 7f5736193700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:46:46.664 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:46:46.665 INFO:tasks.ceph.ceph_manager.ceph:PG 3.8 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:46:51.858 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:46:51.872 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2023-06-09T02:46:51.761104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003029"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":22,"state":"active+undersized+wait","last_fresh":"2023-06-09T02:46:34.700340+0000","last_change":"2023-06-09T02:46:34.699896+0000","last_active":"2023-06-09T02:46:34.700340+0000","last_peered":"2023-06-09T02:46:34.700340+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.700340+0000","last_undegraded":"2023-06-09T02:46:34.700340+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705820,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.707422+0000', 'last_change': '2023-06-09T02:46:47.725692+0000', 'last_active': '2023-06-09T02:46:34.707422+0000', 'last_peered': '2023-06-09T02:46:34.707422+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:46:34.699896+0000', 'last_became_peered': '2023-06-09T02:46:34.699896+0000', 'last_unstale': '2023-06-09T02:46:34.707422+0000', 'last_undegraded': '2023-06-09T02:46:34.707422+0000', 'last_fullsized': '2023-06-09T02:46:34.691990+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:00.020 INFO:tasks.ceph.ceph_manager.ceph:PG 2.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:08.705+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.712+0000 7fed33c4e700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:08.907 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:PG 3.1 is not active+clean , 2023-06-09T02:47:13.383 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.1', 'version': "20'291", 'reported_seq': 787, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.710414+0000', 'last_change': '2023-06-09T02:46:41.715854+0000', 'last_active': '2023-06-09T02:46:34.710414+0000', 'last_peered': '2023-06-09T02:46:34.710414+0000', 'last_clean': '2023-06-09T02:46:33.654852+0000', 'last_became_active': '2023-06-09T02:46:34.696619+0000', 'last_became_peered': '2023-06-09T02:46:34.696619+0000', 'last_unstale': '2023-06-09T02:46:34.710414+0000', 'last_undegraded': '2023-06-09T02:46:34.689238+0000', 'last_fullsized': '2023-06-09T02:46:34.688652+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 291, 'log_dups_size': 0, 'ondisk_log_size': 291, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 143776516, 'num_objects': 48, 'num_object_clones': 0, 'num_object_copies': 96, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 48, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 460, 'num_read_kb': 249496, 'num_write': 700, 'num_write_kb': 201710, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 48}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:13.384 INFO:tasks.ceph.ceph_manager.ceph:PG 3.7 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:47:22.287 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.d', 'version': "20'212", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.703770+0000', 'last_change': '2023-06-09T02:46:41.720448+0000', 'last_active': '2023-06-09T02:46:34.703770+0000', 'last_peered': '2023-06-09T02:46:34.703770+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.696111+0000', 'last_became_peered': '2023-06-09T02:46:34.696111+0000', 'last_unstale': '2023-06-09T02:46:34.703770+0000', 'last_undegraded': '2023-06-09T02:46:34.703770+0000', 'last_fullsized': '2023-06-09T02:46:34.687473+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 212, 'log_dups_size': 0, 'ondisk_log_size': 212, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:PG 3.c is not active+clean , 2023-06-09T02:47:22.288 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.c', 'version': "21'272", 'reported_seq': 205, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692992+0000', 'last_change': '2023-06-09T02:46:25.404450+0000', 'last_active': '2023-06-09T02:46:34.692992+0000', 'last_peered': '2023-06-09T02:46:34.692992+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.718786+0000', 'last_became_peered': '2023-06-09T02:45:52.718786+0000', 'last_unstale': '2023-06-09T02:46:34.692992+0000', 'last_undegraded': '2023-06-09T02:46:25.403808+0000', 'last_fullsized': '2023-06-09T02:45:52.707174+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 272, 'log_dups_size': 0, 'ondisk_log_size': 272, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5111308, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 104749, 'num_write': 162, 'num_write_kb': 49828, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 2}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:26.873+0000 7f4cb1e57700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout: , 2023-06-09T02:47:27.144 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c3a25994-a927-4d0a-bff8-7be9672f8e22","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"2 osds down","count":2},"muted":false},"OSD_HOST_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 host (2 osds) down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 318/932 objects degraded (34.120%), 10 pgs degraded, 13 pgs undersized","count":23},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":203,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":22,"num_osds":4,"num_up_osds":2,"osd_up_since":1686278793,"num_in_osds":4,"osd_in_since":1686278645,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized+degraded","count":10},{"state_name":"active+undersized","count":10},{"state_name":"active+clean","count":5}],"num_pgs":25,"num_pools":3,"num_objects":466,"data_bytes":1375932054,"bytes_used":1566179328,"bytes_avail":427930550272,"bytes_total":429496729600,"degraded_objects":318,"degraded_total":932,"degraded_ratio":0.34120171673819738},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2023-06-09T02:45:11.738171+0000","services":{}},"progress_events":{}} Ignoring malparsed backtrace: 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.8', 'version': "20'359", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.709224+0000', 'last_change': '2023-06-09T02:46:41.747822+0000', 'last_active': '2023-06-09T02:46:34.709224+0000', 'last_peered': '2023-06-09T02:46:34.709224+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.702955+0000', 'last_became_peered': '2023-06-09T02:46:34.702955+0000', 'last_unstale': '2023-06-09T02:46:34.709224+0000', 'last_undegraded': '2023-06-09T02:46:34.709224+0000', 'last_fullsized': '2023-06-09T02:46:34.686747+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 359, 'log_dups_size': 0, 'ondisk_log_size': 359, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:47:35.637 INFO:tasks.ceph.ceph_manager.ceph:PG 3.5 is not active+clean , 2023-06-09T02:47:35.638 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.5', 'version': "20'302", 'reported_seq': 9, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.706704+0000', 'last_change': '2023-06-09T02:46:41.773676+0000', 'last_active': '2023-06-09T02:46:34.706704+0000', 'last_peered': '2023-06-09T02:46:34.706704+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:46:34.701324+0000', 'last_became_peered': '2023-06-09T02:46:34.701324+0000', 'last_unstale': '2023-06-09T02:46:34.706704+0000', 'last_undegraded': '2023-06-09T02:46:34.706704+0000', 'last_fullsized': '2023-06-09T02:46:34.689653+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 302, 'log_dups_size': 0, 'ondisk_log_size': 302, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:47:43.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:47:43.988+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:47:43.994+0000 7f6f7eb97700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:47:48.995 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.714638+0000', 'last_change': '2023-06-09T02:45:59.729905+0000', 'last_active': '2023-06-09T02:45:52.714638+0000', 'last_peered': '2023-06-09T02:45:52.714638+0000', 'last_clean': '2023-06-09T02:44:27.665796+0000', 'last_became_active': '2023-06-09T02:45:52.708834+0000', 'last_became_peered': '2023-06-09T02:45:52.708834+0000', 'last_unstale': '2023-06-09T02:45:52.714638+0000', 'last_undegraded': '2023-06-09T02:45:52.714638+0000', 'last_fullsized': '2023-06-09T02:45:52.703512+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:PG 1.0 is not active+clean , 2023-06-09T02:47:48.996 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '1.0', 'version': "13'97", 'reported_seq': 255, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.692045+0000', 'last_change': '2023-06-09T02:46:05.710796+0000', 'last_active': '2023-06-09T02:46:34.692045+0000', 'last_peered': '2023-06-09T02:46:34.692045+0000', 'last_clean': '2023-06-09T02:44:27.701670+0000', 'last_became_active': '2023-06-09T02:45:52.716766+0000', 'last_became_peered': '2023-06-09T02:45:52.716766+0000', 'last_unstale': '2023-06-09T02:46:34.692045+0000', 'last_undegraded': '2023-06-09T02:45:52.711660+0000', 'last_fullsized': '2023-06-09T02:45:52.711198+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:12.640861+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 1180736, 'num_objects': 4, 'num_object_clones': 0, 'num_object_copies': 8, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 142, 'num_read_kb': 120, 'num_write': 274, 'num_write_kb': 3788, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 4}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} Ignoring malparsed backtrace: dumped all , 2023-06-09T02:47:57.904 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2023-06-09T02:47:57.779026+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} , 2023-06-09T02:47:57.906 INFO:tasks.ceph.ceph_manager.ceph:PG 3.f is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:02.366 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.7', 'version': "20'336", 'reported_seq': 932, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.705105+0000', 'last_change': '2023-06-09T02:46:41.736133+0000', 'last_active': '2023-06-09T02:46:34.705105+0000', 'last_peered': '2023-06-09T02:46:34.705105+0000', 'last_clean': '2023-06-09T02:46:33.628600+0000', 'last_became_active': '2023-06-09T02:46:34.699062+0000', 'last_became_peered': '2023-06-09T02:46:34.699062+0000', 'last_unstale': '2023-06-09T02:46:34.705105+0000', 'last_undegraded': '2023-06-09T02:46:34.692479+0000', 'last_fullsized': '2023-06-09T02:46:34.691837+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 336, 'log_dups_size': 0, 'ondisk_log_size': 336, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 165494196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 583, 'num_read_kb': 325012, 'num_write': 816, 'num_write_kb': 241588, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 60}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:PG 2.6 is not active+clean , 2023-06-09T02:48:02.367 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 29, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682233+0000', 'last_change': '2023-06-09T02:45:59.713810+0000', 'last_active': '2023-06-09T02:46:34.682233+0000', 'last_peered': '2023-06-09T02:46:34.682233+0000', 'last_clean': '2023-06-09T02:44:31.719829+0000', 'last_became_active': '2023-06-09T02:45:52.714365+0000', 'last_became_peered': '2023-06-09T02:45:52.714365+0000', 'last_unstale': '2023-06-09T02:46:34.682233+0000', 'last_undegraded': '2023-06-09T02:46:34.682233+0000', 'last_fullsized': '2023-06-09T02:45:52.708114+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.b is not active+clean , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.b', 'version': "21'242", 'reported_seq': 688, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.679503+0000', 'last_change': '2023-06-09T02:46:01.720164+0000', 'last_active': '2023-06-09T02:46:34.679503+0000', 'last_peered': '2023-06-09T02:46:34.679503+0000', 'last_clean': '2023-06-09T02:45:50.161878+0000', 'last_became_active': '2023-06-09T02:45:52.718038+0000', 'last_became_peered': '2023-06-09T02:45:52.718038+0000', 'last_unstale': '2023-06-09T02:46:34.679503+0000', 'last_undegraded': '2023-06-09T02:45:52.707209+0000', 'last_fullsized': '2023-06-09T02:45:52.706167+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 242, 'log_dups_size': 0, 'ondisk_log_size': 242, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 97926294, 'num_objects': 34, 'num_object_clones': 0, 'num_object_copies': 68, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 34, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 459, 'num_read_kb': 300322, 'num_write': 580, 'num_write_kb': 168476, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [{'shards': '3', 'objects': 34}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} , 2023-06-09T02:48:11.250 INFO:tasks.ceph.ceph_manager.ceph:PG 3.a is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:16.120 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now , 2023-06-09T02:48:16.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2023-06-09T02:48:16.240+0000 7fa1994d9700 -1 WARNING: the following dangerous and experimental features are enabled: crimson Ignoring malparsed backtrace: 2023-06-09T02:48:24.603 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 13, 'reported_epoch': 22, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.682773+0000', 'last_change': '2023-06-09T02:45:59.757634+0000', 'last_active': '2023-06-09T02:46:34.682773+0000', 'last_peered': '2023-06-09T02:46:34.682773+0000', 'last_clean': '2023-06-09T02:44:14.651736+0000', 'last_became_active': '2023-06-09T02:45:52.729380+0000', 'last_became_peered': '2023-06-09T02:45:52.729380+0000', 'last_unstale': '2023-06-09T02:46:34.682773+0000', 'last_undegraded': '2023-06-09T02:46:34.682773+0000', 'last_fullsized': '2023-06-09T02:45:52.718827+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:48:24.604 INFO:tasks.ceph.ceph_manager.ceph:PG 3.3 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:33.153 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all , 2023-06-09T02:48:33.166 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2023-06-09T02:48:31.789080+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1375932054,"num_objects":466,"num_object_clones":0,"num_object_copies":932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":318,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5063,"num_read_kb":2853012,"num_write":7130,"num_write_kb":2014640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4722,"ondisk_log_size":4722,"up":30,"acting":30,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":419430400,"kb_used":1529472,"kb_used_data":1370576,"kb_used_omap":0,"kb_used_meta":140544,"kb_avail":417900928,"statfs":{"total":429496729600,"available":427930550272,"internally_reserved":0,"allocated":1403469824,"data_stored":1394948064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":143917056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003442"},"pg_stats":[{"pgid":"3.f","version":"20'259","reported_seq":112,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.248083+0000","last_change":"2023-06-09T02:46:01.770104+0000","last_active":"2023-06-09T02:46:34.248083+0000","last_peered":"2023-06-09T02:46:34.248083+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707814+0000","last_became_peered":"2023-06-09T02:45:52.707814+0000","last_unstale":"2023-06-09T02:46:34.248083+0000","last_undegraded":"2023-06-09T02:46:34.248083+0000","last_fullsized":"2023-06-09T02:45:52.700919+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3260756,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":108716,"num_write":62,"num_write_kb":18824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"20'310","reported_seq":844,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.687596+0000","last_change":"2023-06-09T02:46:41.724545+0000","last_active":"2023-06-09T02:46:34.687596+0000","last_peered":"2023-06-09T02:46:34.687596+0000","last_clean":"2023-06-09T02:46:32.996280+0000","last_became_active":"2023-06-09T02:46:34.684268+0000","last_became_peered":"2023-06-09T02:46:34.684268+0000","last_unstale":"2023-06-09T02:46:34.687596+0000","last_undegraded":"2023-06-09T02:46:34.680280+0000","last_fullsized":"2023-06-09T02:46:34.679638+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185529196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":485,"num_read_kb":265896,"num_write":770,"num_write_kb":225044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":60}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'212","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.703770+0000","last_change":"2023-06-09T02:46:41.720448+0000","last_active":"2023-06-09T02:46:34.703770+0000","last_peered":"2023-06-09T02:46:34.703770+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.696111+0000","last_became_peered":"2023-06-09T02:46:34.696111+0000","last_unstale":"2023-06-09T02:46:34.703770+0000","last_undegraded":"2023-06-09T02:46:34.703770+0000","last_fullsized":"2023-06-09T02:46:34.687473+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"21'272","reported_seq":205,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692992+0000","last_change":"2023-06-09T02:46:25.404450+0000","last_active":"2023-06-09T02:46:34.692992+0000","last_peered":"2023-06-09T02:46:34.692992+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.718786+0000","last_became_peered":"2023-06-09T02:45:52.718786+0000","last_unstale":"2023-06-09T02:46:34.692992+0000","last_undegraded":"2023-06-09T02:46:25.403808+0000","last_fullsized":"2023-06-09T02:45:52.707174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5111308,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":104749,"num_write":162,"num_write_kb":49828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"21'242","reported_seq":688,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.679503+0000","last_change":"2023-06-09T02:46:01.720164+0000","last_active":"2023-06-09T02:46:34.679503+0000","last_peered":"2023-06-09T02:46:34.679503+0000","last_clean":"2023-06-09T02:45:50.161878+0000","last_became_active":"2023-06-09T02:45:52.718038+0000","last_became_peered":"2023-06-09T02:45:52.718038+0000","last_unstale":"2023-06-09T02:46:34.679503+0000","last_undegraded":"2023-06-09T02:45:52.707209+0000","last_fullsized":"2023-06-09T02:45:52.706167+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97926294,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":34,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":459,"num_read_kb":300322,"num_write":580,"num_write_kb":168476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":34}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"20'236","reported_seq":151,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:33.474100+0000","last_change":"2023-06-09T02:46:10.887913+0000","last_active":"2023-06-09T02:46:33.474100+0000","last_peered":"2023-06-09T02:46:33.474100+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.707238+0000","last_became_peered":"2023-06-09T02:45:52.707238+0000","last_unstale":"2023-06-09T02:46:33.474100+0000","last_undegraded":"2023-06-09T02:46:10.887514+0000","last_fullsized":"2023-06-09T02:45:52.700718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21349840,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":140,"num_read_kb":103816,"num_write":106,"num_write_kb":31122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"20'376","reported_seq":186,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.216213+0000","last_change":"2023-06-09T02:46:34.208943+0000","last_active":"2023-06-09T02:46:34.216213+0000","last_peered":"2023-06-09T02:46:34.216213+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:45:52.711417+0000","last_became_peered":"2023-06-09T02:45:52.711417+0000","last_unstale":"2023-06-09T02:46:34.216213+0000","last_undegraded":"2023-06-09T02:46:34.208643+0000","last_fullsized":"2023-06-09T02:45:52.698894+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4404160,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":107,"num_read_kb":70495,"num_write":154,"num_write_kb":45990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"20'359","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.709224+0000","last_change":"2023-06-09T02:46:41.747822+0000","last_active":"2023-06-09T02:46:34.709224+0000","last_peered":"2023-06-09T02:46:34.709224+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.702955+0000","last_became_peered":"2023-06-09T02:46:34.702955+0000","last_unstale":"2023-06-09T02:46:34.709224+0000","last_undegraded":"2023-06-09T02:46:34.709224+0000","last_fullsized":"2023-06-09T02:46:34.686747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.678243+0000","last_change":"2023-06-09T02:44:15.706766+0000","last_active":"2023-06-09T02:46:34.678243+0000","last_peered":"2023-06-09T02:46:34.678243+0000","last_clean":"2023-06-09T02:46:34.678243+0000","last_became_active":"2023-06-09T02:44:15.703930+0000","last_became_peered":"2023-06-09T02:44:15.703930+0000","last_unstale":"2023-06-09T02:46:34.678243+0000","last_undegraded":"2023-06-09T02:46:34.678243+0000","last_fullsized":"2023-06-09T02:46:34.678243+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"20'302","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.706704+0000","last_change":"2023-06-09T02:46:41.773676+0000","last_active":"2023-06-09T02:46:34.706704+0000","last_peered":"2023-06-09T02:46:34.706704+0000","last_clean":"2023-06-09T02:44:24.632749+0000","last_became_active":"2023-06-09T02:46:34.701324+0000","last_became_peered":"2023-06-09T02:46:34.701324+0000","last_unstale":"2023-06-09T02:46:34.706704+0000","last_undegraded":"2023-06-09T02:46:34.706704+0000","last_fullsized":"2023-06-09T02:46:34.689653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682773+0000","last_change":"2023-06-09T02:45:59.757634+0000","last_active":"2023-06-09T02:46:34.682773+0000","last_peered":"2023-06-09T02:46:34.682773+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.729380+0000","last_became_peered":"2023-06-09T02:45:52.729380+0000","last_unstale":"2023-06-09T02:46:34.682773+0000","last_undegraded":"2023-06-09T02:46:34.682773+0000","last_fullsized":"2023-06-09T02:45:52.718827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"21'313","reported_seq":847,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.680344+0000","last_change":"2023-06-09T02:46:01.731366+0000","last_active":"2023-06-09T02:46:34.680344+0000","last_peered":"2023-06-09T02:46:34.680344+0000","last_clean":"2023-06-09T02:45:51.547062+0000","last_became_active":"2023-06-09T02:45:52.724794+0000","last_became_peered":"2023-06-09T02:45:52.724794+0000","last_unstale":"2023-06-09T02:46:34.680344+0000","last_undegraded":"2023-06-09T02:45:52.707597+0000","last_fullsized":"2023-06-09T02:45:52.706569+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147073196,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":504,"num_read_kb":233526,"num_write":764,"num_write_kb":216182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":52}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.707422+0000","last_change":"2023-06-09T02:46:47.725692+0000","last_active":"2023-06-09T02:46:34.707422+0000","last_peered":"2023-06-09T02:46:34.707422+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:46:34.699896+0000","last_became_peered":"2023-06-09T02:46:34.699896+0000","last_unstale":"2023-06-09T02:46:34.707422+0000","last_undegraded":"2023-06-09T02:46:34.707422+0000","last_fullsized":"2023-06-09T02:46:34.691990+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"20'211","reported_seq":592,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.693775+0000","last_change":"2023-06-09T02:44:25.720469+0000","last_active":"2023-06-09T02:46:34.693775+0000","last_peered":"2023-06-09T02:46:34.693775+0000","last_clean":"2023-06-09T02:46:34.693775+0000","last_became_active":"2023-06-09T02:44:25.717232+0000","last_became_peered":"2023-06-09T02:44:25.717232+0000","last_unstale":"2023-06-09T02:46:34.693775+0000","last_undegraded":"2023-06-09T02:46:34.693775+0000","last_fullsized":"2023-06-09T02:46:34.693775+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118429052,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":364,"num_read_kb":211327,"num_write":518,"num_write_kb":152786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.714638+0000","last_change":"2023-06-09T02:45:59.729905+0000","last_active":"2023-06-09T02:45:52.714638+0000","last_peered":"2023-06-09T02:45:52.714638+0000","last_clean":"2023-06-09T02:44:27.665796+0000","last_became_active":"2023-06-09T02:45:52.708834+0000","last_became_peered":"2023-06-09T02:45:52.708834+0000","last_unstale":"2023-06-09T02:45:52.714638+0000","last_undegraded":"2023-06-09T02:45:52.714638+0000","last_fullsized":"2023-06-09T02:45:52.703512+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'97","reported_seq":255,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.692045+0000","last_change":"2023-06-09T02:46:05.710796+0000","last_active":"2023-06-09T02:46:34.692045+0000","last_peered":"2023-06-09T02:46:34.692045+0000","last_clean":"2023-06-09T02:44:27.701670+0000","last_became_active":"2023-06-09T02:45:52.716766+0000","last_became_peered":"2023-06-09T02:45:52.716766+0000","last_unstale":"2023-06-09T02:46:34.692045+0000","last_undegraded":"2023-06-09T02:45:52.711660+0000","last_fullsized":"2023-06-09T02:45:52.711198+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:12.640861+0000","last_clean_scrub_stamp":"2023-06-09T02:44:12.640861+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"20'256","reported_seq":748,"reported_epoch":20,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.391628+0000","last_change":"2023-06-09T02:46:01.725103+0000","last_active":"2023-06-09T02:46:34.391628+0000","last_peered":"2023-06-09T02:46:34.391628+0000","last_clean":"2023-06-09T02:45:50.291566+0000","last_became_active":"2023-06-09T02:45:52.705375+0000","last_became_peered":"2023-06-09T02:45:52.705375+0000","last_unstale":"2023-06-09T02:46:34.391628+0000","last_undegraded":"2023-06-09T02:45:52.701032+0000","last_fullsized":"2023-06-09T02:45:52.700626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150971298,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":50,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":535,"num_read_kb":343900,"num_write":634,"num_write_kb":185274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":50}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.718119+0000","last_change":"2023-06-09T02:45:59.732836+0000","last_active":"2023-06-09T02:45:52.718119+0000","last_peered":"2023-06-09T02:45:52.718119+0000","last_clean":"2023-06-09T02:44:14.651736+0000","last_became_active":"2023-06-09T02:45:52.713730+0000","last_became_peered":"2023-06-09T02:45:52.713730+0000","last_unstale":"2023-06-09T02:45:52.718119+0000","last_undegraded":"2023-06-09T02:45:52.718119+0000","last_fullsized":"2023-06-09T02:45:52.704955+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"20'291","reported_seq":787,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.710414+0000","last_change":"2023-06-09T02:46:41.715854+0000","last_active":"2023-06-09T02:46:34.710414+0000","last_peered":"2023-06-09T02:46:34.710414+0000","last_clean":"2023-06-09T02:46:33.654852+0000","last_became_active":"2023-06-09T02:46:34.696619+0000","last_became_peered":"2023-06-09T02:46:34.696619+0000","last_unstale":"2023-06-09T02:46:34.710414+0000","last_undegraded":"2023-06-09T02:46:34.689238+0000","last_fullsized":"2023-06-09T02:46:34.688652+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143776516,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":48,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":460,"num_read_kb":249496,"num_write":700,"num_write_kb":201710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":48}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.686770+0000","last_change":"2023-06-09T02:44:15.716230+0000","last_active":"2023-06-09T02:46:34.686770+0000","last_peered":"2023-06-09T02:46:34.686770+0000","last_clean":"2023-06-09T02:46:34.686770+0000","last_became_active":"2023-06-09T02:44:15.713088+0000","last_became_peered":"2023-06-09T02:44:15.713088+0000","last_unstale":"2023-06-09T02:46:34.686770+0000","last_undegraded":"2023-06-09T02:46:34.686770+0000","last_fullsized":"2023-06-09T02:46:34.686770+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"20'258","reported_seq":696,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.681485+0000","last_change":"2023-06-09T02:44:25.729880+0000","last_active":"2023-06-09T02:46:34.681485+0000","last_peered":"2023-06-09T02:46:34.681485+0000","last_clean":"2023-06-09T02:46:34.681485+0000","last_became_active":"2023-06-09T02:44:25.724912+0000","last_became_peered":"2023-06-09T02:44:25.724912+0000","last_unstale":"2023-06-09T02:46:34.681485+0000","last_undegraded":"2023-06-09T02:46:34.681485+0000","last_fullsized":"2023-06-09T02:46:34.681485+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136517278,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":396,"num_read_kb":202661,"num_write":636,"num_write_kb":190824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"20'336","reported_seq":932,"reported_epoch":22,"state":"active+undersized+degraded","last_fresh":"2023-06-09T02:46:34.705105+0000","last_change":"2023-06-09T02:46:41.736133+0000","last_active":"2023-06-09T02:46:34.705105+0000","last_peered":"2023-06-09T02:46:34.705105+0000","last_clean":"2023-06-09T02:46:33.628600+0000","last_became_active":"2023-06-09T02:46:34.699062+0000","last_became_peered":"2023-06-09T02:46:34.699062+0000","last_unstale":"2023-06-09T02:46:34.705105+0000","last_undegraded":"2023-06-09T02:46:34.692479+0000","last_fullsized":"2023-06-09T02:46:34.691837+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165494196,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":60,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":583,"num_read_kb":325012,"num_write":816,"num_write_kb":241588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[{"shards":"3","objects":60}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":22,"state":"active+undersized","last_fresh":"2023-06-09T02:46:34.682233+0000","last_change":"2023-06-09T02:45:59.713810+0000","last_active":"2023-06-09T02:46:34.682233+0000","last_peered":"2023-06-09T02:46:34.682233+0000","last_clean":"2023-06-09T02:44:31.719829+0000","last_became_active":"2023-06-09T02:45:52.714365+0000","last_became_peered":"2023-06-09T02:45:52.714365+0000","last_unstale":"2023-06-09T02:46:34.682233+0000","last_undegraded":"2023-06-09T02:46:34.682233+0000","last_fullsized":"2023-06-09T02:45:52.708114+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":20,"state":"active+undersized","last_fresh":"2023-06-09T02:45:52.717013+0000","last_change":"2023-06-09T02:45:59.717605+0000","last_active":"2023-06-09T02:45:52.717013+0000","last_peered":"2023-06-09T02:45:52.717013+0000","last_clean":"2023-06-09T02:44:27.664689+0000","last_became_active":"2023-06-09T02:45:52.709685+0000","last_became_peered":"2023-06-09T02:45:52.709685+0000","last_unstale":"2023-06-09T02:45:52.717013+0000","last_undegraded":"2023-06-09T02:45:52.717013+0000","last_fullsized":"2023-06-09T02:45:52.702540+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:14.651736+0000","last_clean_scrub_stamp":"2023-06-09T02:44:14.651736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"20'392","reported_seq":1043,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-06-09T02:46:34.679039+0000","last_change":"2023-06-09T02:44:25.710225+0000","last_active":"2023-06-09T02:46:34.679039+0000","last_peered":"2023-06-09T02:46:34.679039+0000","last_clean":"2023-06-09T02:46:34.679039+0000","last_became_active":"2023-06-09T02:44:25.706609+0000","last_became_peered":"2023-06-09T02:44:25.706609+0000","last_unstale":"2023-06-09T02:46:34.679039+0000","last_undegraded":"2023-06-09T02:46:34.679039+0000","last_fullsized":"2023-06-09T02:46:34.679039+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-06-09T02:44:24.632749+0000","last_clean_scrub_stamp":"2023-06-09T02:44:24.632749+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194908228,"num_objects":64,"num_object_clones":0,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":623,"num_read_kb":332976,"num_write":954,"num_write_kb":283204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1374751318,"num_objects":462,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":314,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4921,"num_read_kb":2852892,"num_write":6856,"num_write_kb":2010852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4625,"ondisk_log_size":4625,"up":19,"acting":19,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":1180736,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":120,"num_write":274,"num_write_kb":3788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":10,"num_store_stats":0}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":453404,"kb_used_data":406952,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104404196,"statfs":{"total":107374182400,"available":106909896704,"internally_reserved":0,"allocated":416718848,"data_stored":414204176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":255172,"kb_used_data":219996,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104602428,"statfs":{"total":107374182400,"available":107112886272,"internally_reserved":0,"allocated":225275904,"data_stored":223886957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":326352,"kb_used_data":292692,"kb_used_omap":0,"kb_used_meta":31040,"kb_avail":104531248,"statfs":{"total":107374182400,"available":107039997952,"internally_reserved":0,"allocated":299716608,"data_stored":297828855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31784960},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":494544,"kb_used_data":450936,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":104363056,"statfs":{"total":107374182400,"available":106867769344,"internally_reserved":0,"allocated":461758464,"data_stored":459028076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} Ignoring malparsed backtrace: 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 3.2 is not active+clean , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.2', 'version': "20'256", 'reported_seq': 748, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.391628+0000', 'last_change': '2023-06-09T02:46:01.725103+0000', 'last_active': '2023-06-09T02:46:34.391628+0000', 'last_peered': '2023-06-09T02:46:34.391628+0000', 'last_clean': '2023-06-09T02:45:50.291566+0000', 'last_became_active': '2023-06-09T02:45:52.705375+0000', 'last_became_peered': '2023-06-09T02:45:52.705375+0000', 'last_unstale': '2023-06-09T02:46:34.391628+0000', 'last_undegraded': '2023-06-09T02:45:52.701032+0000', 'last_fullsized': '2023-06-09T02:45:52.700626+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 256, 'log_dups_size': 0, 'ondisk_log_size': 256, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 150971298, 'num_objects': 50, 'num_object_clones': 0, 'num_object_copies': 100, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 50, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 535, 'num_read_kb': 343900, 'num_write': 634, 'num_write_kb': 185274, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 50}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:37.957 INFO:tasks.ceph.ceph_manager.ceph:PG 2.0 is not active+clean Ignoring malparsed backtrace: 2023-06-09T02:48:46.860 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.f', 'version': "20'259", 'reported_seq': 112, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:46:34.248083+0000', 'last_change': '2023-06-09T02:46:01.770104+0000', 'last_active': '2023-06-09T02:46:34.248083+0000', 'last_peered': '2023-06-09T02:46:34.248083+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707814+0000', 'last_became_peered': '2023-06-09T02:45:52.707814+0000', 'last_unstale': '2023-06-09T02:46:34.248083+0000', 'last_undegraded': '2023-06-09T02:46:34.248083+0000', 'last_fullsized': '2023-06-09T02:45:52.700919+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 259, 'log_dups_size': 0, 'ondisk_log_size': 259, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 3260756, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 125, 'num_read_kb': 108716, 'num_write': 62, 'num_write_kb': 18824, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:PG 3.e is not active+clean , 2023-06-09T02:48:46.861 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.e', 'version': "20'310", 'reported_seq': 844, 'reported_epoch': 22, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.687596+0000', 'last_change': '2023-06-09T02:46:41.724545+0000', 'last_active': '2023-06-09T02:46:34.687596+0000', 'last_peered': '2023-06-09T02:46:34.687596+0000', 'last_clean': '2023-06-09T02:46:32.996280+0000', 'last_became_active': '2023-06-09T02:46:34.684268+0000', 'last_became_peered': '2023-06-09T02:46:34.684268+0000', 'last_unstale': '2023-06-09T02:46:34.687596+0000', 'last_undegraded': '2023-06-09T02:46:34.680280+0000', 'last_fullsized': '2023-06-09T02:46:34.679638+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 310, 'log_dups_size': 0, 'ondisk_log_size': 310, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 185529196, 'num_objects': 60, 'num_object_clones': 0, 'num_object_copies': 120, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 60, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 485, 'num_read_kb': 265896, 'num_write': 770, 'num_write_kb': 225044, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 60}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:48:51.326 INFO:tasks.ceph.ceph_manager.ceph:PG 2.7 is not active+clean , 2023-06-09T02:48:51.327 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 20, 'state': 'active+undersized', 'last_fresh': '2023-06-09T02:45:52.717013+0000', 'last_change': '2023-06-09T02:45:59.717605+0000', 'last_active': '2023-06-09T02:45:52.717013+0000', 'last_peered': '2023-06-09T02:45:52.717013+0000', 'last_clean': '2023-06-09T02:44:27.664689+0000', 'last_became_active': '2023-06-09T02:45:52.709685+0000', 'last_became_peered': '2023-06-09T02:45:52.709685+0000', 'last_unstale': '2023-06-09T02:45:52.717013+0000', 'last_undegraded': '2023-06-09T02:45:52.717013+0000', 'last_fullsized': '2023-06-09T02:45:52.702540+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:48:51.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json Ignoring malparsed backtrace: 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.a', 'version': "20'236", 'reported_seq': 151, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:33.474100+0000', 'last_change': '2023-06-09T02:46:10.887913+0000', 'last_active': '2023-06-09T02:46:33.474100+0000', 'last_peered': '2023-06-09T02:46:33.474100+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.707238+0000', 'last_became_peered': '2023-06-09T02:45:52.707238+0000', 'last_unstale': '2023-06-09T02:46:33.474100+0000', 'last_undegraded': '2023-06-09T02:46:10.887514+0000', 'last_fullsized': '2023-06-09T02:45:52.700718+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 236, 'log_dups_size': 0, 'ondisk_log_size': 236, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 21349840, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 12, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 6, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 140, 'num_read_kb': 103816, 'num_write': 106, 'num_write_kb': 31122, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 6}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} , 2023-06-09T02:49:00.227 INFO:tasks.ceph.ceph_manager.ceph:PG 3.9 is not active+clean , 2023-06-09T02:49:00.228 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.9', 'version': "20'376", 'reported_seq': 186, 'reported_epoch': 20, 'state': 'active+undersized+degraded', 'last_fresh': '2023-06-09T02:46:34.216213+0000', 'last_change': '2023-06-09T02:46:34.208943+0000', 'last_active': '2023-06-09T02:46:34.216213+0000', 'last_peered': '2023-06-09T02:46:34.216213+0000', 'last_clean': '2023-06-09T02:44:24.632749+0000', 'last_became_active': '2023-06-09T02:45:52.711417+0000', 'last_became_peered': '2023-06-09T02:45:52.711417+0000', 'last_unstale': '2023-06-09T02:46:34.216213+0000', 'last_undegraded': '2023-06-09T02:46:34.208643+0000', 'last_fullsized': '2023-06-09T02:45:52.698894+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 376, 'log_dups_size': 0, 'ondisk_log_size': 376, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4404160, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 107, 'num_read_kb': 70495, 'num_write': 154, 'num_write_kb': 45990, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:05.221+0000 7f4524fc6700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:05.414 INFO:teuthology.orchestra.run.smithi119.stdout: , dumped all Ignoring malparsed backtrace: 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '3.3', 'version': "21'313", 'reported_seq': 849, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.782700+0000', 'last_change': '2023-06-09T02:49:11.782700+0000', 'last_active': '2023-06-09T02:49:11.782700+0000', 'last_peered': '2023-06-09T02:46:34.681801+0000', 'last_clean': '2023-06-09T02:45:51.547062+0000', 'last_became_active': '2023-06-09T02:45:52.724794+0000', 'last_became_peered': '2023-06-09T02:45:52.724794+0000', 'last_unstale': '2023-06-09T02:49:11.782700+0000', 'last_undegraded': '2023-06-09T02:49:11.782700+0000', 'last_fullsized': '2023-06-09T02:49:11.782700+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 15, 'last_epoch_clean': 16, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:24.632749+0000', 'objects_scrubbed': 0, 'log_size': 313, 'log_dups_size': 0, 'ondisk_log_size': 313, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 147073196, 'num_objects': 52, 'num_object_clones': 0, 'num_object_copies': 104, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 504, 'num_read_kb': 233526, 'num_write': 764, 'num_write_kb': 216182, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [0], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:PG 2.1 is not active+clean , 2023-06-09T02:49:13.606 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 25, 'reported_epoch': 23, 'state': 'peering', 'last_fresh': '2023-06-09T02:49:11.813946+0000', 'last_change': '2023-06-09T02:49:11.743688+0000', 'last_active': '2023-06-09T02:49:11.743688+0000', 'last_peered': '2023-06-09T02:45:52.702176+0000', 'last_clean': '2023-06-09T02:45:52.702176+0000', 'last_became_active': '2023-06-09T02:44:15.722764+0000', 'last_became_peered': '2023-06-09T02:44:15.722764+0000', 'last_unstale': '2023-06-09T02:49:11.813946+0000', 'last_undegraded': '2023-06-09T02:49:11.813946+0000', 'last_fullsized': '2023-06-09T02:49:11.813946+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 12, 'last_epoch_clean': 13, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'last_clean_scrub_stamp': '2023-06-09T02:44:14.651736+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} Ignoring malparsed backtrace: 2023-06-09T02:49:27.085+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.092+0000 7fe78c664700 -1 WARNING: the following dangerous and experimental features are enabled: crimson , 2023-06-09T02:49:27.287 INFO:teuthology.orchestra.run.smithi119.stdout: Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_teuthology_2ef0dcdee569e9851c9d2ffb6acfd4b80d031e61/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 65, in reraise Ignoring malparsed backtrace: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.106366.log.gz , /var/log/ceph/ceph-client.admin.107963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110663.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107963.log.gz Ignoring malparsed backtrace: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.115335.log.gz , /var/log/ceph/ceph-client.admin.117001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112240.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.117001.log.gz Ignoring malparsed backtrace: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109781.log , /var/log/ceph/ceph-client.admin.107596.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.107596.log.gz , /var/log/ceph/ceph-client.admin.114053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107794.log Ignoring malparsed backtrace: /var/log/ceph/ceph-client.admin.108330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113465.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.108330.log.gz , gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116756.log: /var/log/ceph/ceph-client.admin.112191.log Ignoring malparsed backtrace: gzip /var/log/ceph/ceph-client.admin.110173.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.112338.log , 50.0% -- replaced with /var/log/ceph/ceph-client.admin.110173.log.gz , /var/log/ceph/ceph-client.admin.109340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111056.log Ignoring malparsed backtrace: 2023-06-09T02:50:19.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2023-06-09T02:50:19.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2023-06-09T02:50:19.028 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz Found 7 distinct failure reasons Dead: b'2023-06-09T14:11:38.151 INFO:teuthology.orchestra.run.smithi114.stdout:{"epoch":1062,"fsid":"ddf0ff98-d2dc-4292-b9b3-c218cf5e4302","created":"2023-06-09T02:38:37.487921+0000","modified":"2023-06-09T02:56:42.721942+0000","last_up_change":"2023-06-09T02:39:20.695348+0000","last_in_change":"2023-06-09T02:56:37.720236+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-09T02:38:45.965939+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0\'0","target_version":"0\'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-09T02:38:48.103544+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0\'0","target_version":"0\'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":0.75,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e4a3cb3-98ab-4442-a117-eee30338f945","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1751801000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6801","nonce":1751801000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1751801000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6805","nonce":1751801000}]},"public_addr":"172.21.15.114:6800/1751801000","cluster_addr":"172.21.15.114:6801/1751801000","heartbeat_back_addr":"172.21.15.114:6804/1751801000","heartbeat_front_addr":"172.21.15.114:6805/1751801000","state":["exists","up"]},{"osd":1,"uuid":"57a5770d-f4bb-427d-8bfb-ad3ea982afc4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":16,"down_at":39,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6803","nonce":922609838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":922609838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6807","nonce":922609838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":922609838}]},"public_addr":"172.21.15.114:6803/922609838","cluster_addr":"172.21.15.114:6802/922609838","heartbeat_back_addr":"172.21.15.114:6807/922609838","heartbeat_front_addr":"172.21.15.114:6806/922609838","state":["autoout","exists"]},{"osd":2,"uuid":"2af16818-8d33-4300-b7f9-029341a6ecaa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6801","nonce":4193893684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4193893684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4193893684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6807","nonce":4193893684}]},"public_addr":"172.21.15.192:6801/4193893684","cluster_addr":"172.21.15.192:6800/4193893684","heartbeat_back_addr":"172.21.15.192:6806/4193893684","heartbeat_front_addr":"172.21.15.192:6807/4193893684","state":["exists","up"]},{"osd":3,"uuid":"4faa1c3d-73f2-4b2e-9529-50c89bb60b0a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":1057,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2605666393}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6803","nonce":2605666393}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6809","nonce":2605666393}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2605666393}]},"public_addr":"172.21.15.192:6802/2605666393","cluster_addr":"172.21.15.192:6803/2605666393","heartbeat_back_addr":"172.21.15.192:6809/2605666393","heartbeat_front_addr":"172.21.15.192:6808/2605666393","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"2023-06-09T02:39:20.695348+0000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":1,"length":2},{"begin":4,"length":1},{"begin":7,"length":3},{"begin":13,"length":1},{"begin":15,"length":2},{"begin":19,"length":1},{"begin":21,"length":4},{"begin":26,"length":1},{"begin":30,"length":2},{"begin":35,"length":3},{"begin":39,"length":1},{"begin":41,"length":2},{"begin":44,"length":1},{"begin":48,"length":1},{"begin":50,"length":2},{"begin":54,"length":1},{"begin":56,"length":3},{"begin":60,"length":2},{"begin":63,"length":1},{"begin":67,"length":4},{"begin":74,"length":1},{"begin":76,"length":4},{"begin":81,"length":2},{"begin":85,"length":1},{"begin":88,"length":1},{"begin":90,"length":2},{"begin":93,"length":6},{"begin":100,"length":3},{"begin":105,"length":1},{"begin":107,"length":1},{"begin":110,"length":2},{"begin":113,"length":3},{"begin":117,"length":2},{"begin":120,"length":2},{"begin":123,"length":1},{"begin":134,"length":2},{"begin":137,"length":2},{"begin":141,"length":1},{"begin":144,"length":2},{"begin":147,"length":1},{"begin":149,"length":3},{"begin":159,"length":1},{"begin":161,"length":1},{"begin":164,"length":1},{"begin":168,"length":1},{"begin":170,"length":1},{"begin":172,"length":1},{"begin":175,"length":1},{"begin":177,"length":3},{"begin":183,"length":2},{"begin":187,"length":1},{"begin":189,"length":1},{"begin":191,"length":1},{"begin":193,"length":2},{"begin":197,"length":2},{"begin":201,"length":3},{"begin":207,"length":4},{"begin":214,"length":1},{"begin":217,"length":4},{"begin":225,"length":1},{"begin":227,"length":3},{"begin":232,"length":5},{"begin":239,"length":1},{"begin":242,"length":1},{"begin":245,"length":1},{"begin":247,"length":2},{"begin":250,"length":1},{"begin":252,"length":1},{"begin":255,"length":4},{"begin":263,"length":1},{"begin":265,"length":1},{"begin":267,"length":3},{"begin":271,"length":1},{"begin":275,"length":1},{"begin":277,"length":3},{"begin":281,"length":2},{"begin":287,"length":1},{"begin":290,"length":1},{"begin":293,"length":3},{"begin":297,"length":2},{"begin":300,"length":2},{"begin":304,"length":1},{"begin":306,"length":2},{"begin":312,"length":3},{"begin":316,"length":1},{"begin":318,"length":3},{"begin":322,"length":1},{"begin":325,"length":1},{"begin":328,"length":2},{"begin":332,"length":1},{"begin":335,"length":2},{"begin":338,"length":7},{"begin":347,"length":1},{"begin":351,"length":1},{"begin":354,"length":2},{"begin":357,"length":3},{"begin":362,"length":1},{"begin":366,"length":4},{"begin":372,"length":2},{"begin":378,"length":1},{"begin":382,"length":3},{"begin":386,"length":2},{"begin":389,"length":1},{"begin":391,"length":1},{"begin":394,"length":4},{"begin":402,"length":1},{"begin":405,"length":1},{"begin":410,"length":1},{"begin":412,"length":1},{"begin":414,"length":1},{"begin":416,"length":1},{"begin":418,"length":1},{"begin":420,"length":2},{"begin":424,"length":3},{"begin":429,"length":1},{"begin":431,"length":2},{"begin":434,"length":4},{"begin":441,"length":1},{"begin":447,"length":4},{"begin":452,"length":2},{"begin":455,"length":1},{"begin":459,"length":1},{"begin":463,"length":1},{"begin":465,"length":2},{"begin":469,"length":2},{"begin":472,"length":3},{"begin":476,"length":1},{"begin":478,"length":2},{"begin":482,"length":1},{"begin":486,"length":1},{"begin":488,"length":1},{"begin":491,"length":5},{"begin":498,"length":3},{"begin":502,"length":1},{"begin":504,"length":2},{"begin":507,"length":2},{"begin":511,"length":1},{"begin":514,"length":1},{"begin":517,"length":3},{"begin":522,"length":1},{"begin":525,"length":3},{"begin":534,"length":4},{"begin":539,"length":1},{"begin":542,"length":1},{"begin":544,"length":3},{"begin":548,"length":8},{"begin":557,"length":1},{"begin":560,"length":2},{"begin":563,"length":2},{"begin":567,"length":2},{"begin":570,"length":2},{"begin":573,"length":1},{"begin":577,"length":3},{"begin":581,"length":4},{"begin":586,"length":1},{"begin":588,"length":1},{"begin":592,"length":1},{"begin":599,"length":1},{"begin":603,"length":1},{"begin":605,"length":5},{"begin":611,"length":1},{"begin":613,"length":1},{"begin":616,"length":1},{"begin":624,"length":3},{"begin":628,"length":3},{"begin":633,"length":5},{"begin":639,"length":4},{"begin":645,"length":2},{"begin":649,"length":1},{"begin":653,"length":3},{"begin":657,"length":1},{"begin":661,"length":5},{"begin":670,"length":2},{"begin":675,"length":6},{"begin":683,"length":1},{"begin":685,"length":1},{"begin":689,"length":1},{"begin":697,"length":1},{"begin":700,"length":1},{"begin":703,"length":1},{"begin":706,"length":1},{"begin":708,"length":2},{"begin":711,"length":1},{"begin":713,"length":4},{"begin":718,"length":1},{"begin":720,"length":1},{"begin":722,"length":1},{"begin":724,"length":4},{"begin":729,"length":1},{"begin":732,"length":1},{"begin":734,"length":1},{"begin":736,"length":3},{"begin":741,"length":2},{"begin":745,"length":8},{"begin":754,"length":4},{"begin":759,"length":2},{"begin":763,"length":1},{"begin":765,"length":1},{"begin":767,"length":2},{"begin":771,"length":1},{"begin":775,"length":1},{"begin":785,"length":2},{"begin":791,"length":1},{"begin":794,"length":1},{"begin":796,"length":3},{"begin":800,"length":1},{"begin":803,"length":1},{"begin":805,"length":1},{"begin":810,"length":1},{"begin":812,"length":1},{"begin":816,"length":2},{"begin":819,"length":2},{"begin":822,"length":1},{"begin":824,"length":5},{"begin":831,"length":1},{"begin":835,"length":2},{"begin":838,"length":1},{"begin":840,"length":1},{"begin":843,"length":4},{"begin":848,"length":4},{"begin":854,"length":1},{"begin":856,"length":2},{"begin":859,"length":1},{"begin":866,"length":1},{"begin":868,"length":1},{"begin":870,"length":6},{"begin":877,"length":1},{"begin":880,"length":1},{"begin":884,"length":5},{"begin":890,"length":2},{"begin":893,"length":2},{"begin":897,"length":4},{"begin":907,"length":1},{"begin":909,"length":2},{"begin":915,"length":1},{"begin":919,"length":1},{"begin":923,"length":1},{"begin":925,"length":4},{"begin":930,"length":1},{"begin":933,"length":2},{"begin":936,"length":2},{"begin":940,"length":6},{"begin":950,"length":1},{"begin":955,"length":1},{"begin":958,"length":2},{"begin":962,"length":1},{"begin":964,"length":4},{"begin":972,"length":1},{"begin":974,"length":1},{"begin":979,"length":2},{"begin":982,"length":1},{"begin":984,"length":1},{"begin":988,"length":1},{"begin":990,"length":2},{"begin":1002,"length":3},{"begin":1007,"length":2},{"begin":1015,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}}' 2 jobs: ['7299699', '7299702'] suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/pool-snaps-few-objects}', 'workloads/radosbench}'] Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2 jobs: ['7299697', '7299700'] suites intersection: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] suites union: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 1 jobs: ['7299706'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-localized}'] Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 1 jobs: ['7299696'] suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0' 1 jobs: ['7299710'] suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/readwrite}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh --eval-attr \ 1 jobs: ['7299711'] suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests}'] Failure: Command failed on smithi026 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 1 jobs: ['7299712'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/snaps-few-objects-localized}']