2022-02-03T03:53:57.287 INFO:root:teuthology version: 1.1.0-0f634dc1 2022-02-03T03:53:57.288 INFO:teuthology.results:subset: '111/120000' 2022-02-03T03:53:57.288 INFO:teuthology.results:seed: '5382' 2022-02-03T06:36:57.501 INFO:root:teuthology version: 1.1.0-0f634dc1 2022-02-03T06:36:57.532 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-02-03T12:50:32.823 INFO:teuthology.results:Tests finished! gathering results... 2022-02-03T12:51:10.318 INFO:scrape:Found 327 jobs 2022-02-03T12:51:20.028 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:20.029 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:20.030 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.031 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.032 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.033 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.034 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:20.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.035 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.036 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.037 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.038 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.039 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:20.040 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:20.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:20.041 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:20.042 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:20.043 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.044 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:20.045 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:20.046 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:20.047 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:20.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.048 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.049 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:20.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:20.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:20.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:20.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:20.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:20.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:20.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:20.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:20.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:20.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:20.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:20.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.082 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:20.084 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.088 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:20.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:20.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:20.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:20.093 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:20.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:20.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:20.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:20.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:20.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:20.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:20.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:20.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:20.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:20.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:20.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:20.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:20.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:20.117 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.118 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:20.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:20.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:20.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:20.129 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:20.130 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:20.131 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:20.131 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:20.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.133 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.134 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.135 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.136 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.137 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:20.140 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:20.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:20.150 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:20.156 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:20.158 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:20.159 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:20.159 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:20.160 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:20.300 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:20.301 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:20.302 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.303 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.304 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:20.305 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.306 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.306 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.307 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.308 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.308 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.309 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:20.310 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:20.311 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:20.311 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:20.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:20.312 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:20.314 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:20.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:20.315 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:20.316 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.317 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:20.318 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.319 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.320 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.321 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.322 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:20.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.323 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.324 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.325 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:20.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.326 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:20.327 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:20.328 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:20.328 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:20.329 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:20.329 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:20.330 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.331 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.332 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.332 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:20.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:20.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:20.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:20.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:20.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:20.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:20.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:20.342 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:20.342 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:20.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:20.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:20.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:20.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:20.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:20.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:20.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:20.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:20.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.354 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:20.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:20.359 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:20.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:20.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.367 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.367 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:20.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:20.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:20.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:20.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:20.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:20.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:20.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:20.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:20.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:20.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:20.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:20.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:20.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:20.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:20.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:20.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:20.542 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:20.543 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:20.544 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.545 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.545 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.546 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.547 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:20.547 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.548 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.549 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.549 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.550 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.551 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:20.552 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:20.553 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:20.554 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:20.554 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:20.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.555 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:20.556 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:20.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:20.557 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:20.558 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.558 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.559 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:20.560 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.562 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.563 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.564 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.565 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:20.566 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.567 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.568 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:20.569 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:20.570 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:20.571 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:20.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:20.572 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:20.573 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:20.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.574 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.575 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.575 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.576 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:20.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:20.577 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:20.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.578 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.579 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:20.580 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.581 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.582 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:20.583 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:20.584 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:20.585 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:20.586 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:20.586 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:20.587 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:20.588 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.588 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:20.589 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:20.590 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:20.591 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.591 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:20.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:20.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:20.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:20.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:20.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:20.603 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:20.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:20.605 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.609 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.613 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:20.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:20.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:20.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:20.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:20.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:20.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:20.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:20.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.624 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:20.625 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:20.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:20.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:20.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:20.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:20.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:20.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:20.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:20.784 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:20.786 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:20.787 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.788 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.789 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.790 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:20.791 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.792 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.793 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:20.796 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.797 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:20.798 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:20.799 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:20.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:20.800 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:20.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.801 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:20.802 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:20.803 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:20.804 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:20.805 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.806 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.808 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:20.809 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.810 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.811 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.812 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.813 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.814 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.816 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:20.817 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.819 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.820 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.821 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:20.822 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:20.823 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:20.824 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:20.825 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:20.827 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:20.828 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:20.829 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:20.830 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:20.831 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:20.832 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.833 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.834 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:20.836 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:20.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:20.837 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:20.839 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.840 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.841 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.842 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:20.843 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.844 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.845 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.846 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.847 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:20.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:20.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:20.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:20.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:20.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:20.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:20.853 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:20.855 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:20.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:20.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:20.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:20.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:20.864 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:20.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:20.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.866 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:20.866 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.867 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.868 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:20.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:20.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:20.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:20.872 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:20.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:20.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:20.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:20.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:20.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:20.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:20.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:20.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:20.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:20.886 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:20.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:20.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:20.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:20.891 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:20.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:20.892 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:20.897 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:20.902 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:20.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:20.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:20.910 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:20.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:20.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:21.049 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:21.050 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:21.051 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.052 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.053 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:21.054 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.055 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.055 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.056 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.057 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.058 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:21.059 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:21.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:21.060 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:21.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:21.061 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.062 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:21.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:21.063 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:21.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:21.064 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.065 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:21.066 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.067 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.068 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.069 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:21.070 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.071 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.072 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.073 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:21.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.074 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:21.075 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:21.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:21.076 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:21.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:21.077 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:21.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.078 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.079 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.080 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:21.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:21.081 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:21.082 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.083 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.084 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:21.084 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.085 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.086 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:21.087 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:21.088 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:21.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:21.089 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:21.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:21.090 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:21.091 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.092 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:21.093 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:21.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:21.094 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.095 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.096 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.097 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.098 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:21.099 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:21.100 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:21.100 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:21.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.101 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.102 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.103 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:21.104 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.105 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:21.106 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:21.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.107 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:21.108 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.109 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.110 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.111 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.112 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.113 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.114 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.115 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.116 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.117 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:21.117 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:21.118 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:21.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:21.119 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:21.120 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:21.121 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:21.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:21.122 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.123 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.124 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.125 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.126 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.127 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:21.128 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:21.132 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:21.138 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:21.143 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:21.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:21.145 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:21.146 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:21.147 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:21.332 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:21.333 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:21.334 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.335 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.336 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:21.337 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.338 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.339 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.340 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.341 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.342 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:21.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:21.343 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:21.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:21.344 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:21.345 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.346 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:21.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:21.347 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:21.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:21.348 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.349 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:21.350 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.351 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.352 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.353 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:21.355 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.356 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.357 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:21.358 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.359 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:21.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:21.360 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:21.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:21.361 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:21.362 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:21.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.363 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.364 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.365 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:21.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:21.366 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:21.367 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.368 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:21.369 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.370 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.371 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:21.372 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:21.373 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:21.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:21.374 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:21.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:21.375 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:21.376 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.377 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:21.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:21.378 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:21.379 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.380 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.381 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.382 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.383 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:21.384 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:21.385 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:21.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:21.386 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.387 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.387 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.388 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.389 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.389 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:21.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.390 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.391 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:21.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:21.392 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:21.393 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.394 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.395 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.395 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.396 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.397 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.398 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.399 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.400 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.401 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.402 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.402 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:21.403 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:21.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:21.404 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:21.405 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:21.406 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:21.407 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:21.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:21.408 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.409 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.410 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.411 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.412 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.413 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:21.414 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:21.418 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:21.424 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:21.430 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:21.432 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:21.432 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:21.433 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:21.434 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:21.591 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:21.592 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:21.593 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.594 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.595 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:21.596 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.597 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.598 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.599 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.600 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:21.601 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:21.602 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:21.603 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:21.604 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:21.606 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.607 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:21.608 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:21.610 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:21.611 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:21.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.612 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.614 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:21.615 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.616 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.617 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.618 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:21.619 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.620 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.621 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:21.622 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.623 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:21.626 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:21.627 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:21.628 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:21.630 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:21.631 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:21.632 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.633 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.634 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.635 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.636 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:21.638 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:21.639 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:21.640 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.641 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.642 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.643 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:21.644 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.646 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.647 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:21.648 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:21.649 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:21.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:21.650 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:21.651 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:21.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:21.652 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.653 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:21.654 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:21.655 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:21.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.656 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.657 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.658 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.659 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.660 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:21.661 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:21.661 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:21.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:21.662 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.663 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.665 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:21.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.669 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.670 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:21.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:21.671 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:21.672 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.673 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.674 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.675 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.676 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.677 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.678 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.679 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.680 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:21.681 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:21.682 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:21.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:21.683 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:21.684 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:21.685 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:21.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:21.686 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.687 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.688 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.689 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.690 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.691 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:21.692 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:21.696 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:21.701 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:21.707 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:21.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:21.709 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:21.710 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:21.711 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:21.847 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-10314-gfc4f59a1 (fc4f59a13b5cfe31b1d75dc7e873c108d04bff91) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f9ec5e4bf64] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x283185) [0x7f9ec5e4c185] 2022-02-03T12:51:21.848 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "MOSDPGPush(4.d 166/165 [PushOp(4:b1c761f9:::smithi09033386-13:52, version: 163'118, data_included: [1675490~872745,2844123~63521], data_size: 936266, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(4:b1c761f9:::smithi09033386-13:52@163'118, size: 2907644, copy_subset: [0~2907644], clone_subset: {}, snapset: 67=[]:{3d=[2e],52=[52],67=[67,66,64,63,61,60,5a,56,54]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2907644, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1675490, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:11:53.693582+0000", , 2022-02-03T04:12:07.953 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 14.244802862, 2022-02-03T12:51:21.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4610000000000002e-06 , 2022-02-03T04:12:07.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:07.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.849 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091866+0000", , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0363000000000001e-05 , 2022-02-03T04:12:08.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.850 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:08.143 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.148044+0000", , 2022-02-03T04:12:08.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.851 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:08.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.347677+0000", 2022-02-03T12:51:21.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:08.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.852 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.854 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.857 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.858 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2022-02-03T12:51:21.859 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.999999 , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:08.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.860 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:08.383 INFO:teuthology.orchestra.run.smithi160.stdout:} , 2022-02-03T04:12:08.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2022-02-03T12:51:21.861 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4394", , 2022-02-03T04:12:08.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", 2022-02-03T12:51:21.862 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2022-02-03T12:51:21.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:10:57.129708+0000", , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.758565727000004, , 2022-02-03T04:12:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.030533663999999999, 2022-02-03T12:51:21.863 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:57.159997+0000", 2022-02-03T12:51:21.865 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.869 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:53.486132+0000", , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.402141873, , 2022-02-03T04:12:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027021092, 2022-02-03T12:51:21.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513153+0000", 2022-02-03T12:51:21.870 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", , 2022-02-03T04:12:09.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513191+0000", 2022-02-03T12:51:21.871 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", , 2022-02-03T04:12:09.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:53.513584+0000", 2022-02-03T12:51:21.873 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.115 INFO:teuthology.orchestra.run.smithi090.stdout: ] , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.874 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.138 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4531 4.b 4.c3d07c6b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2022-02-03T04:11:57.680679+0000", , 2022-02-03T04:12:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.207594289999999, 2022-02-03T12:51:21.875 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0223e-05 , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.876 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0093750670000000008 , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012792500000000001 , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.877 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131507 , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1380000000000007e-06 , 2022-02-03T04:12:09.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.878 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027832600000000002 , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.879 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ , 2022-02-03T04:12:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:1214 4.4 4:22a5f7cc:::smithi09033386-19:head [write 645402~551297] snapc e=[d,5,4,2] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e43)", 2022-02-03T12:51:21.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.025740587999999998, , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { , 2022-02-03T04:12:09.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.880 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1216 , 2022-02-03T04:12:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.881 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237742+0000", , 2022-02-03T04:12:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237760+0000", , 2022-02-03T04:12:09.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2022-02-03T12:51:21.882 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:42.237842+0000", , 2022-02-03T04:12:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2022-02-03T12:51:21.883 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: } , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2022-02-03T12:51:21.884 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.288848+0000", 2022-02-03T12:51:21.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289215+0000", 2022-02-03T12:51:21.885 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289479+0000", 2022-02-03T12:51:21.886 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289508+0000", 2022-02-03T12:51:21.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:09:47.289529+0000", 2022-02-03T12:51:21.887 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:10:35.608249+0000", 2022-02-03T12:51:21.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2022-02-03T12:51:21.888 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.184 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.889 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.890 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.231 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2022-02-03T12:51:21.891 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:10.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2022-02-03T12:51:21.900 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2022-02-03T12:51:21.902 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.707 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { , 2022-02-03T04:12:10.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", 2022-02-03T12:51:21.903 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.904 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00050517800000000001 , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.904 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031426700000000002 , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:10.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.905 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", , 2022-02-03T04:12:10.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2022-02-03T12:51:21.906 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.857752+0000", , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999976 , 2022-02-03T04:12:10.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.906 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.230514+0000", , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016135100000000001 , 2022-02-03T04:12:10.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.907 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:34.236257+0000", , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2038000000000003e-05 , 2022-02-03T04:12:10.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:53.356081+0000", , 2022-02-03T04:12:10.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.908 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:22.444201+0000", , 2022-02-03T04:12:10.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0070842350000000004 2022-02-03T12:51:21.909 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:10.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116050+0000", 2022-02-03T12:51:21.911 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:10.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:10.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.116766+0000", 2022-02-03T12:51:21.912 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:11.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.117725+0000", 2022-02-03T12:51:21.913 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", , 2022-02-03T04:12:11.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:49.118539+0000", 2022-02-03T12:51:21.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4394", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:11.069 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4296 2022-02-03T12:51:21.914 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.474 INFO:teuthology.orchestra.run.smithi090.stdout:4.4 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.17 , 2022-02-03T04:12:11.475 INFO:teuthology.orchestra.run.smithi090.stdout:4.14 2022-02-03T12:51:21.915 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.814919+0000", , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0068661119999999997 , 2022-02-03T04:12:11.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.018230751, , 2022-02-03T04:12:11.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { , 2022-02-03T04:12:11.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2022-02-03T12:51:21.916 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.818397+0000", , 2022-02-03T04:12:11.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4552000000000001e-05 2022-02-03T12:51:21.917 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:24.861584+0000", 2022-02-03T12:51:21.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2022-02-03T04:09:24.865698+0000", , 2022-02-03T04:12:11.685 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.67146077699999, , 2022-02-03T04:12:11.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016890948999999999, 2022-02-03T12:51:21.918 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:11.740 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:11.741 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.919 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8652000000000002e-05 , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.920 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2179e-05 , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3132 , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.923 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124253+0000", , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3889999999999998e-06 , 2022-02-03T04:12:11.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:11.890 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:11.891 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4394.0:4554 4.a 4:586ade6a:::smithi09033386-47:head [read 0~3848836,checksum] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.924 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:11.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.124330+0000", , 2022-02-03T04:12:11.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9260000000000002e-06 2022-02-03T12:51:21.925 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", , 2022-02-03T04:12:12.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.430827+0000", 2022-02-03T12:51:21.926 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 0,6", , 2022-02-03T04:12:12.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.945332+0000", 2022-02-03T12:51:21.926 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.404 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", , 2022-02-03T04:12:12.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:09:07.953337+0000", 2022-02-03T12:51:21.927 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: ] , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.451 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2022-02-03T12:51:21.928 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 , 2022-02-03T04:12:12.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.929 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8121999999999995e-05 , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.498 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.929 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103553 , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.930 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5104999999999997e-05 , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.544 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2022-02-03T12:51:21.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:55.126833+0000", , 2022-02-03T04:12:12.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7979999999999999e-06 , 2022-02-03T04:12:12.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.931 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.138835+0000", , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003272184 , 2022-02-03T04:12:12.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.933 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:10:57.117784+0000", , 2022-02-03T04:12:12.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4294967295.9999995 2022-02-03T12:51:21.934 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 3,6", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:00.133726+0000", , 2022-02-03T04:12:12.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000189502 2022-02-03T12:51:21.935 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: } , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:12.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:12.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", , 2022-02-03T04:12:12.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:36.401701+0000", 2022-02-03T12:51:21.936 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.119505+0000", , 2022-02-03T04:12:13.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0550000000000001e-05 , 2022-02-03T04:12:13.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.937 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.313754+0000", , 2022-02-03T04:12:13.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.190594755 , 2022-02-03T04:12:13.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.316127+0000", , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000203389 , 2022-02-03T04:12:13.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.938 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.317712+0000", , 2022-02-03T04:12:13.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023261500000000001 , 2022-02-03T04:12:13.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.939 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:14.322230+0000", , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117087 , 2022-02-03T04:12:13.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.940 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.168293+0000", , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5115000000000001e-05 , 2022-02-03T04:12:13.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.940 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177481+0000", , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0419000000000005e-05 , 2022-02-03T04:12:13.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.941 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177360+0000", , 2022-02-03T04:12:13.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0064976119999999998 , 2022-02-03T04:12:13.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.171732+0000", , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011505099999999999 , 2022-02-03T04:12:13.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.942 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4538 , 2022-02-03T04:12:13.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.943 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.338 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/3468573123", , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4539 , 2022-02-03T04:12:13.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.945 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4540 4.14 4:2984f270:::smithi09033386-16:head [setxattr _header (57) in=7b,truncate 2420274] snapc 6b=[6b,6a,67,66,64,63,61,60,56,52,2e] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.945 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.177130+0000", , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7114999999999996e-05 , 2022-02-03T04:12:13.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2022-02-03T12:51:21.946 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4394.0:4546 4.14 4:2dd53d5e:::smithi09033386-45:head [read 0~2285785,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e172)", 2022-02-03T12:51:21.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", , 2022-02-03T04:12:13.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127748+0000", , 2022-02-03T04:12:13.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.616e-06 2022-02-03T12:51:21.947 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.153598+0000", , 2022-02-03T04:12:13.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109166 2022-02-03T12:51:21.948 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", , 2022-02-03T04:12:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.127801+0000", 2022-02-03T12:51:21.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", , 2022-02-03T04:12:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2022-02-03T04:11:59.128076+0000", 2022-02-03T12:51:21.949 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.506366+0000", , 2022-02-03T04:12:13.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0010464000000000001 , 2022-02-03T04:12:13.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2022-02-03T12:51:21.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.958 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", , 2022-02-03T04:12:13.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:53.779973+0000", , 2022-02-03T04:12:13.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6824000000000007e-05 2022-02-03T12:51:21.950 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", , 2022-02-03T04:12:13.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:54.091856+0000", 2022-02-03T12:51:21.951 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ , 2022-02-03T04:12:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2022-02-03T12:51:21.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { , 2022-02-03T04:12:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2022-02-03T12:51:21.952 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3587000000000001e-05 , 2022-02-03T04:12:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.953 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5571999999999998e-05 , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2022-02-03T12:51:21.954 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4521 , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4522 , 2022-02-03T04:12:14.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.128 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.956 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4523 , 2022-02-03T04:12:14.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.152 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4524 , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, , 2022-02-03T04:12:14.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2022-02-03T12:51:21.957 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.139426+0000", , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5132e-05 , 2022-02-03T04:12:14.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2022-02-03T12:51:21.958 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2022-02-03T04:11:59.161837+0000", , 2022-02-03T04:12:14.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2022-02-03T12:51:21.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:14.801 INFO:teuthology.orchestra.run.smithi090.stdout:setting '_remove' omap key , 2022-02-03T04:12:14.805 INFO:teuthology.orchestra.run.smithi090.stdout:finish_remove_pgs 4.6_head removing 4.6 , 2022-02-03T04:12:14.812 INFO:teuthology.orchestra.run.smithi090.stdout:Remove successful 2022-02-03T12:51:21.959 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:37.488+0000 7f2ed9c28700 -1 mon.a@0(leader).osd e198 definitely_dead 0 , marked in osd.2. , 2022-02-03T04:12:38.188 INFO:tasks.thrashosds.thrasher:Added osd 2 2022-02-03T12:51:21.964 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:12:58.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 , 2022-02-03T04:12:58.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 , 2022-02-03T04:12:58.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-02-03T12:51:21.971 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:16.810 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2022-02-03T04:13:16.340296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":508463744,"num_objects":323,"num_object_clones":119,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":322,"num_whiteouts":7,"num_read":3019,"num_read_kb":2926866,"num_write":1705,"num_write_kb":500265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":892237064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":117,"acting":117,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":1225656,"kb_used_data":1034584,"kb_used_omap":56,"kb_used_meta":190471,"kb_avail":753749064,"statfs":{"total":773094113280,"available":771839041536,"internally_reserved":0,"allocated":1059414016,"data_stored":1135184085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58238,"internal_metadata":195042434},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-20,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":28060854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.267069"},"pg_stats":[{"pgid":"4.19","version":"205'233","reported_seq":945,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141800+0000","last_change":"2022-02-03T04:12:45.100892+0000","last_active":"2022-02-03T04:13:10.141800+0000","last_peered":"2022-02-03T04:13:10.141800+0000","last_clean":"2022-02-03T04:13:10.141800+0000","last_became_active":"2022-02-03T04:12:42.134619+0000","last_became_peered":"2022-02-03T04:12:42.134619+0000","last_unstale":"2022-02-03T04:13:10.141800+0000","last_undegraded":"2022-02-03T04:13:10.141800+0000","last_fullsized":"2022-02-03T04:13:10.141800+0000","mapping_epoch":202,"log_start":"186'225","ondisk_log_start":"186'225","created":111,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'233","last_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_deep_scrub":"205'233","last_deep_scrub_stamp":"2022-02-03T04:12:45.100755+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.100755+0000","objects_scrubbed":8,"log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:04.270829+0000","stat_sum":{"num_bytes":17271325,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":82530,"num_write":31,"num_write_kb":9337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":31554836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"195'175","reported_seq":834,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135862+0000","last_change":"2022-02-03T04:12:41.137126+0000","last_active":"2022-02-03T04:13:10.135862+0000","last_peered":"2022-02-03T04:13:10.135862+0000","last_clean":"2022-02-03T04:13:10.135862+0000","last_became_active":"2022-02-03T04:12:41.136747+0000","last_became_peered":"2022-02-03T04:12:41.136747+0000","last_unstale":"2022-02-03T04:13:10.135862+0000","last_undegraded":"2022-02-03T04:13:10.135862+0000","last_fullsized":"2022-02-03T04:13:10.135862+0000","mapping_epoch":201,"log_start":"194'168","ondisk_log_start":"194'168","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'173","last_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_deep_scrub":"194'173","last_deep_scrub_stamp":"2022-02-03T04:12:32.115187+0000","last_clean_scrub_stamp":"2022-02-03T04:12:32.115187+0000","objects_scrubbed":5,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.697380+0000","stat_sum":{"num_bytes":12214460,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7730451,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.17","version":"188'98","reported_seq":728,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149210+0000","last_change":"2022-02-03T04:13:11.149210+0000","last_active":"2022-02-03T04:13:11.149210+0000","last_peered":"2022-02-03T04:13:11.149210+0000","last_clean":"2022-02-03T04:13:11.149210+0000","last_became_active":"2022-02-03T04:13:11.148591+0000","last_became_peered":"2022-02-03T04:13:11.148591+0000","last_unstale":"2022-02-03T04:13:11.149210+0000","last_undegraded":"2022-02-03T04:13:11.149210+0000","last_fullsized":"2022-02-03T04:13:11.149210+0000","mapping_epoch":225,"log_start":"185'94","ondisk_log_start":"185'94","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"185'94","last_scrub_stamp":"2022-02-03T04:12:17.020146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.020146+0000","objects_scrubbed":4,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.330713+0000","stat_sum":{"num_bytes":8551091,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":7336921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"217'285","reported_seq":1158,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135881+0000","last_change":"2022-02-03T04:12:59.099778+0000","last_active":"2022-02-03T04:13:10.135881+0000","last_peered":"2022-02-03T04:13:10.135881+0000","last_clean":"2022-02-03T04:13:10.135881+0000","last_became_active":"2022-02-03T04:11:54.123983+0000","last_became_peered":"2022-02-03T04:11:54.123983+0000","last_unstale":"2022-02-03T04:13:10.135881+0000","last_undegraded":"2022-02-03T04:13:10.135881+0000","last_fullsized":"2022-02-03T04:13:10.135881+0000","mapping_epoch":166,"log_start":"180'276","ondisk_log_start":"180'276","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'285","last_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_deep_scrub":"217'285","last_deep_scrub_stamp":"2022-02-03T04:12:59.099658+0000","last_clean_scrub_stamp":"2022-02-03T04:12:59.099658+0000","objects_scrubbed":9,"log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.104326+0000","stat_sum":{"num_bytes":18410394,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":49,"num_read_kb":76007,"num_write":9,"num_write_kb":3256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":29393252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":262,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139951+0000","last_change":"2022-02-03T04:13:07.931058+0000","last_active":"2022-02-03T04:13:10.139951+0000","last_peered":"2022-02-03T04:13:10.139951+0000","last_clean":"2022-02-03T04:13:10.139951+0000","last_became_active":"2022-02-03T04:12:39.139890+0000","last_became_peered":"2022-02-03T04:12:39.139890+0000","last_unstale":"2022-02-03T04:13:10.139951+0000","last_undegraded":"2022-02-03T04:13:10.139951+0000","last_fullsized":"2022-02-03T04:13:10.139951+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:07.930967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.930967+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:37.336284+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":405,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.904847+0000","last_change":"2022-02-03T04:13:11.904847+0000","last_active":"2022-02-03T04:13:11.904847+0000","last_peered":"2022-02-03T04:13:11.904847+0000","last_clean":"2022-02-03T04:13:11.904847+0000","last_became_active":"2022-02-03T04:10:54.114267+0000","last_became_peered":"2022-02-03T04:10:54.114267+0000","last_unstale":"2022-02-03T04:13:11.904847+0000","last_undegraded":"2022-02-03T04:13:11.904847+0000","last_fullsized":"2022-02-03T04:13:11.904847+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:11.904743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:13:11.904743+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138871+0000","last_change":"2022-02-03T04:12:38.170543+0000","last_active":"2022-02-03T04:13:10.138871+0000","last_peered":"2022-02-03T04:13:10.138871+0000","last_clean":"2022-02-03T04:13:10.138871+0000","last_became_active":"2022-02-03T04:12:38.170192+0000","last_became_peered":"2022-02-03T04:12:38.170192+0000","last_unstale":"2022-02-03T04:13:10.138871+0000","last_undegraded":"2022-02-03T04:13:10.138871+0000","last_fullsized":"2022-02-03T04:13:10.138871+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:26.975483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.975483+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:49.003000+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145921+0000","last_change":"2022-02-03T04:12:58.163728+0000","last_active":"2022-02-03T04:13:10.145921+0000","last_peered":"2022-02-03T04:13:10.145921+0000","last_clean":"2022-02-03T04:13:10.145921+0000","last_became_active":"2022-02-03T04:11:52.474762+0000","last_became_peered":"2022-02-03T04:11:52.474762+0000","last_unstale":"2022-02-03T04:13:10.145921+0000","last_undegraded":"2022-02-03T04:13:10.145921+0000","last_fullsized":"2022-02-03T04:13:10.145921+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:58.163634+0000","last_clean_scrub_stamp":"2022-02-03T04:12:58.163634+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:19.923361+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"188'233","reported_seq":877,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138004+0000","last_change":"2022-02-03T04:13:00.263146+0000","last_active":"2022-02-03T04:13:10.138004+0000","last_peered":"2022-02-03T04:13:10.138004+0000","last_clean":"2022-02-03T04:13:10.138004+0000","last_became_active":"2022-02-03T04:12:19.138479+0000","last_became_peered":"2022-02-03T04:12:19.138479+0000","last_unstale":"2022-02-03T04:13:10.138004+0000","last_undegraded":"2022-02-03T04:13:10.138004+0000","last_fullsized":"2022-02-03T04:13:10.138004+0000","mapping_epoch":187,"log_start":"170'227","ondisk_log_start":"170'227","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'233","last_scrub_stamp":"2022-02-03T04:13:00.263058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:00.263058+0000","objects_scrubbed":7,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:00.540542+0000","stat_sum":{"num_bytes":10856363,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":33070,"num_write":15,"num_write_kb":4467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12111448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'182","reported_seq":858,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137716+0000","last_change":"2022-02-03T04:12:14.273852+0000","last_active":"2022-02-03T04:13:10.137716+0000","last_peered":"2022-02-03T04:13:10.137716+0000","last_clean":"2022-02-03T04:13:10.137716+0000","last_became_active":"2022-02-03T04:12:14.134184+0000","last_became_peered":"2022-02-03T04:12:14.134184+0000","last_unstale":"2022-02-03T04:13:10.137716+0000","last_undegraded":"2022-02-03T04:13:10.137716+0000","last_fullsized":"2022-02-03T04:13:10.137716+0000","mapping_epoch":185,"log_start":"180'179","ondisk_log_start":"180'179","created":19,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'182","last_scrub_stamp":"2022-02-03T04:12:14.273776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.273776+0000","objects_scrubbed":40,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.641847+0000","stat_sum":{"num_bytes":40170136,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":151,"num_read_kb":11516,"num_write":192,"num_write_kb":82707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":99722828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":278,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.146045+0000","last_change":"2022-02-03T04:12:54.073934+0000","last_active":"2022-02-03T04:13:10.146045+0000","last_peered":"2022-02-03T04:13:10.146045+0000","last_clean":"2022-02-03T04:13:10.146045+0000","last_became_active":"2022-02-03T04:12:38.173512+0000","last_became_peered":"2022-02-03T04:12:38.173512+0000","last_unstale":"2022-02-03T04:13:10.146045+0000","last_undegraded":"2022-02-03T04:13:10.146045+0000","last_fullsized":"2022-02-03T04:13:10.146045+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:54.073816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.073816+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.854243+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"194'297","reported_seq":1072,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137892+0000","last_change":"2022-02-03T04:13:09.334393+0000","last_active":"2022-02-03T04:13:10.137892+0000","last_peered":"2022-02-03T04:13:10.137892+0000","last_clean":"2022-02-03T04:12:34.138484+0000","last_became_active":"2022-02-03T04:12:39.156972+0000","last_became_peered":"2022-02-03T04:12:39.156972+0000","last_unstale":"2022-02-03T04:13:10.137892+0000","last_undegraded":"2022-02-03T04:13:10.137892+0000","last_fullsized":"2022-02-03T04:13:10.137892+0000","mapping_epoch":199,"log_start":"180'292","ondisk_log_start":"180'292","created":20,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'297","last_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_deep_scrub":"194'297","last_deep_scrub_stamp":"2022-02-03T04:12:31.444474+0000","last_clean_scrub_stamp":"2022-02-03T04:12:31.444474+0000","objects_scrubbed":6,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.194586+0000","stat_sum":{"num_bytes":12807971,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":67,"num_read_kb":99592,"num_write":18,"num_write_kb":5711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":19641257,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":6}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"175'183","reported_seq":709,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141624+0000","last_change":"2022-02-03T04:12:38.967384+0000","last_active":"2022-02-03T04:13:10.141624+0000","last_peered":"2022-02-03T04:13:10.141624+0000","last_clean":"2022-02-03T04:13:10.141624+0000","last_became_active":"2022-02-03T04:11:49.124572+0000","last_became_peered":"2022-02-03T04:11:49.124572+0000","last_unstale":"2022-02-03T04:13:10.141624+0000","last_undegraded":"2022-02-03T04:13:10.141624+0000","last_fullsized":"2022-02-03T04:13:10.141624+0000","mapping_epoch":161,"log_start":"172'180","ondisk_log_start":"172'180","created":19,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'183","last_scrub_stamp":"2022-02-03T04:12:38.967274+0000","last_deep_scrub":"139'152","last_deep_scrub_stamp":"2022-02-03T04:11:25.937656+0000","last_clean_scrub_stamp":"2022-02-03T04:12:38.967274+0000","objects_scrubbed":42,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:56.752019+0000","stat_sum":{"num_bytes":39030313,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":121,"num_read_kb":6618,"num_write":166,"num_write_kb":68242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":57751167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":398,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145949+0000","last_change":"2022-02-03T04:12:44.068835+0000","last_active":"2022-02-03T04:13:10.145949+0000","last_peered":"2022-02-03T04:13:10.145949+0000","last_clean":"2022-02-03T04:13:10.145949+0000","last_became_active":"2022-02-03T04:09:07.403158+0000","last_became_peered":"2022-02-03T04:09:07.403158+0000","last_unstale":"2022-02-03T04:13:10.145949+0000","last_undegraded":"2022-02-03T04:13:10.145949+0000","last_fullsized":"2022-02-03T04:13:10.145949+0000","mapping_epoch":15,"log_start":"17'114","ondisk_log_start":"17'114","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'117","last_scrub_stamp":"2022-02-03T04:12:44.068729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:06.383715+0000","last_clean_scrub_stamp":"2022-02-03T04:12:44.068729+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.944660+0000","stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":112,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145093+0000","last_change":"2022-02-03T04:13:02.356908+0000","last_active":"2022-02-03T04:13:10.145093+0000","last_peered":"2022-02-03T04:13:10.145093+0000","last_clean":"2022-02-03T04:13:10.145093+0000","last_became_active":"2022-02-03T04:12:39.139320+0000","last_became_peered":"2022-02-03T04:12:39.139320+0000","last_unstale":"2022-02-03T04:13:10.145093+0000","last_undegraded":"2022-02-03T04:13:10.145093+0000","last_fullsized":"2022-02-03T04:13:10.145093+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:13:02.356806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:11:49.553851+0000","last_clean_scrub_stamp":"2022-02-03T04:13:02.356806+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:26.019027+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"107'169","reported_seq":757,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139593+0000","last_change":"2022-02-03T04:12:35.990568+0000","last_active":"2022-02-03T04:13:10.139593+0000","last_peered":"2022-02-03T04:13:10.139593+0000","last_clean":"2022-02-03T04:13:10.139593+0000","last_became_active":"2022-02-03T04:11:55.125056+0000","last_became_peered":"2022-02-03T04:11:55.125056+0000","last_unstale":"2022-02-03T04:13:10.139593+0000","last_undegraded":"2022-02-03T04:13:10.139593+0000","last_fullsized":"2022-02-03T04:13:10.139593+0000","mapping_epoch":167,"log_start":"107'164","ondisk_log_start":"107'164","created":20,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'169","last_scrub_stamp":"2022-02-03T04:12:35.990452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:35.990452+0000","objects_scrubbed":0,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:36.076686+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"183'130","reported_seq":726,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138242+0000","last_change":"2022-02-03T04:12:44.196151+0000","last_active":"2022-02-03T04:13:10.138242+0000","last_peered":"2022-02-03T04:13:10.138242+0000","last_clean":"2022-02-03T04:13:10.138242+0000","last_became_active":"2022-02-03T04:12:44.195551+0000","last_became_peered":"2022-02-03T04:12:44.195551+0000","last_unstale":"2022-02-03T04:13:10.138242+0000","last_undegraded":"2022-02-03T04:13:10.138242+0000","last_fullsized":"2022-02-03T04:13:10.138242+0000","mapping_epoch":204,"log_start":"175'128","ondisk_log_start":"175'128","created":19,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'125","last_scrub_stamp":"2022-02-03T04:11:49.779647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:11:49.779647+0000","objects_scrubbed":32,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.261841+0000","stat_sum":{"num_bytes":34432940,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":103,"num_read_kb":8013,"num_write":134,"num_write_kb":52373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":73426044,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.143015+0000","last_change":"2022-02-03T04:12:39.146865+0000","last_active":"2022-02-03T04:13:10.143015+0000","last_peered":"2022-02-03T04:13:10.143015+0000","last_clean":"2022-02-03T04:13:10.143015+0000","last_became_active":"2022-02-03T04:12:39.146486+0000","last_became_peered":"2022-02-03T04:12:39.146486+0000","last_unstale":"2022-02-03T04:13:10.143015+0000","last_undegraded":"2022-02-03T04:13:10.143015+0000","last_fullsized":"2022-02-03T04:13:10.143015+0000","mapping_epoch":199,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:15.983756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:15.983756+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:40.159624+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"181'104","reported_seq":708,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148677+0000","last_change":"2022-02-03T04:13:11.148677+0000","last_active":"2022-02-03T04:13:11.148677+0000","last_peered":"2022-02-03T04:13:11.148677+0000","last_clean":"2022-02-03T04:13:11.148677+0000","last_became_active":"2022-02-03T04:13:11.148224+0000","last_became_peered":"2022-02-03T04:13:11.148224+0000","last_unstale":"2022-02-03T04:13:11.148677+0000","last_undegraded":"2022-02-03T04:13:11.148677+0000","last_fullsized":"2022-02-03T04:13:11.148677+0000","mapping_epoch":225,"log_start":"179'102","ondisk_log_start":"179'102","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'104","last_scrub_stamp":"2022-02-03T04:12:29.543391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:29.543391+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.770709+0000","stat_sum":{"num_bytes":7062399,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":25290,"num_write":9,"num_write_kb":2376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":11208687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"175'149","reported_seq":552,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136154+0000","last_change":"2022-02-03T04:12:54.218069+0000","last_active":"2022-02-03T04:13:10.136154+0000","last_peered":"2022-02-03T04:13:10.136154+0000","last_clean":"2022-02-03T04:13:10.136154+0000","last_became_active":"2022-02-03T04:11:55.121746+0000","last_became_peered":"2022-02-03T04:11:55.121746+0000","last_unstale":"2022-02-03T04:13:10.136154+0000","last_undegraded":"2022-02-03T04:13:10.136154+0000","last_fullsized":"2022-02-03T04:13:10.136154+0000","mapping_epoch":167,"log_start":"168'146","ondisk_log_start":"168'146","created":19,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'149","last_scrub_stamp":"2022-02-03T04:12:54.217969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:16.082766+0000","last_clean_scrub_stamp":"2022-02-03T04:12:54.217969+0000","objects_scrubbed":36,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:22.632049+0000","stat_sum":{"num_bytes":31785668,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":117,"num_read_kb":2973,"num_write":157,"num_write_kb":50040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":33520724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":102,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145937+0000","last_change":"2022-02-03T04:12:38.172034+0000","last_active":"2022-02-03T04:13:10.145937+0000","last_peered":"2022-02-03T04:13:10.145937+0000","last_clean":"2022-02-03T04:13:10.145937+0000","last_became_active":"2022-02-03T04:12:38.171675+0000","last_became_peered":"2022-02-03T04:12:38.171675+0000","last_unstale":"2022-02-03T04:13:10.145937+0000","last_undegraded":"2022-02-03T04:13:10.145937+0000","last_fullsized":"2022-02-03T04:13:10.145937+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:12:14.683074+0000","last_clean_scrub_stamp":"2022-02-03T04:12:14.683074+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:19.978125+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"203'265","reported_seq":969,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138317+0000","last_change":"2022-02-03T04:12:45.781188+0000","last_active":"2022-02-03T04:13:10.138317+0000","last_peered":"2022-02-03T04:13:10.138317+0000","last_clean":"2022-02-03T04:13:10.138317+0000","last_became_active":"2022-02-03T04:12:40.136567+0000","last_became_peered":"2022-02-03T04:12:40.136567+0000","last_unstale":"2022-02-03T04:13:10.138317+0000","last_undegraded":"2022-02-03T04:13:10.138317+0000","last_fullsized":"2022-02-03T04:13:10.138317+0000","mapping_epoch":200,"log_start":"175'261","ondisk_log_start":"175'261","created":20,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'265","last_scrub_stamp":"2022-02-03T04:12:45.781046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.781046+0000","objects_scrubbed":9,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:46.223218+0000","stat_sum":{"num_bytes":21383399,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":81150,"num_write":15,"num_write_kb":4365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"174'87","reported_seq":682,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137770+0000","last_change":"2022-02-03T04:13:09.333252+0000","last_active":"2022-02-03T04:13:10.137770+0000","last_peered":"2022-02-03T04:13:10.137770+0000","last_clean":"2022-02-03T04:11:50.760433+0000","last_became_active":"2022-02-03T04:12:39.150593+0000","last_became_peered":"2022-02-03T04:12:39.150593+0000","last_unstale":"2022-02-03T04:13:10.137770+0000","last_undegraded":"2022-02-03T04:13:10.137770+0000","last_fullsized":"2022-02-03T04:13:10.137770+0000","mapping_epoch":199,"log_start":"154'84","ondisk_log_start":"154'84","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'87","last_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_deep_scrub":"174'87","last_deep_scrub_stamp":"2022-02-03T04:12:21.608806+0000","last_clean_scrub_stamp":"2022-02-03T04:12:21.608806+0000","objects_scrubbed":2,"log_size":3,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":6263240,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":100571,"num_write":24,"num_write_kb":7341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":14523222,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[0,5,4],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.138226+0000","last_change":"2022-02-03T04:12:38.171216+0000","last_active":"2022-02-03T04:13:10.138226+0000","last_peered":"2022-02-03T04:13:10.138226+0000","last_clean":"2022-02-03T04:13:10.138226+0000","last_became_active":"2022-02-03T04:12:38.170863+0000","last_became_peered":"2022-02-03T04:12:38.170863+0000","last_unstale":"2022-02-03T04:13:10.138226+0000","last_undegraded":"2022-02-03T04:13:10.138226+0000","last_fullsized":"2022-02-03T04:13:10.138226+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:17.265686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:07.390875+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.265686+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:26.841481+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"180'207","reported_seq":786,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145076+0000","last_change":"2022-02-03T04:12:28.523251+0000","last_active":"2022-02-03T04:13:10.145076+0000","last_peered":"2022-02-03T04:13:10.145076+0000","last_clean":"2022-02-03T04:13:10.145076+0000","last_became_active":"2022-02-03T04:11:49.126299+0000","last_became_peered":"2022-02-03T04:11:49.126299+0000","last_unstale":"2022-02-03T04:13:10.145076+0000","last_undegraded":"2022-02-03T04:13:10.145076+0000","last_fullsized":"2022-02-03T04:13:10.145076+0000","mapping_epoch":161,"log_start":"170'203","ondisk_log_start":"170'203","created":20,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'207","last_scrub_stamp":"2022-02-03T04:12:28.523125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.523125+0000","objects_scrubbed":7,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:51.257952+0000","stat_sum":{"num_bytes":15015147,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":33,"num_read_kb":39125,"num_write":15,"num_write_kb":3897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":7292544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"106'132","reported_seq":761,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.136037+0000","last_change":"2022-02-03T04:13:01.828747+0000","last_active":"2022-02-03T04:13:10.136037+0000","last_peered":"2022-02-03T04:13:10.136037+0000","last_clean":"2022-02-03T04:12:34.136525+0000","last_became_active":"2022-02-03T04:13:01.827636+0000","last_became_peered":"2022-02-03T04:13:01.827636+0000","last_unstale":"2022-02-03T04:13:10.136037+0000","last_undegraded":"2022-02-03T04:13:10.136037+0000","last_fullsized":"2022-02-03T04:13:10.136037+0000","mapping_epoch":219,"log_start":"104'128","ondisk_log_start":"104'128","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'132","last_scrub_stamp":"2022-02-03T04:11:35.694822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:11:35.694822+0000","objects_scrubbed":0,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,3,6],"avail_no_missing":["4","1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"179'246","reported_seq":943,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.149610+0000","last_change":"2022-02-03T04:13:11.149610+0000","last_active":"2022-02-03T04:13:11.149610+0000","last_peered":"2022-02-03T04:13:11.149610+0000","last_clean":"2022-02-03T04:13:11.149610+0000","last_became_active":"2022-02-03T04:13:11.149105+0000","last_became_peered":"2022-02-03T04:13:11.149105+0000","last_unstale":"2022-02-03T04:13:11.149610+0000","last_undegraded":"2022-02-03T04:13:11.149610+0000","last_fullsized":"2022-02-03T04:13:11.149610+0000","mapping_epoch":225,"log_start":"163'239","ondisk_log_start":"163'239","created":20,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'246","last_scrub_stamp":"2022-02-03T04:13:09.942797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:09.942797+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:13.272102+0000","stat_sum":{"num_bytes":16985933,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":312,"num_read_kb":383815,"num_write":108,"num_write_kb":29533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":43843094,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"181'222","reported_seq":984,"reported_epoch":225,"state":"active+remapped+backfill_toofull","last_fresh":"2022-02-03T04:13:10.137748+0000","last_change":"2022-02-03T04:13:09.333972+0000","last_active":"2022-02-03T04:13:10.137748+0000","last_peered":"2022-02-03T04:13:10.137748+0000","last_clean":"2022-02-03T04:12:34.138355+0000","last_became_active":"2022-02-03T04:12:39.156898+0000","last_became_peered":"2022-02-03T04:12:39.156898+0000","last_unstale":"2022-02-03T04:13:10.137748+0000","last_undegraded":"2022-02-03T04:13:10.137748+0000","last_fullsized":"2022-02-03T04:13:10.137748+0000","mapping_epoch":199,"log_start":"180'216","ondisk_log_start":"180'216","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"175'213","last_scrub_stamp":"2022-02-03T04:12:03.157062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:03.157062+0000","objects_scrubbed":12,"log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:23.800825+0000","stat_sum":{"num_bytes":26100611,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":240,"num_read_kb":269804,"num_write":125,"num_write_kb":33732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":84512801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":12}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"191'207","reported_seq":896,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.135993+0000","last_change":"2022-02-03T04:12:30.198470+0000","last_active":"2022-02-03T04:13:10.135993+0000","last_peered":"2022-02-03T04:13:10.135993+0000","last_clean":"2022-02-03T04:13:10.135993+0000","last_became_active":"2022-02-03T04:12:26.185680+0000","last_became_peered":"2022-02-03T04:12:26.185680+0000","last_unstale":"2022-02-03T04:13:10.135993+0000","last_undegraded":"2022-02-03T04:13:10.135993+0000","last_fullsized":"2022-02-03T04:13:10.135993+0000","mapping_epoch":190,"log_start":"191'194","ondisk_log_start":"191'194","created":20,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'205","last_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_deep_scrub":"191'205","last_deep_scrub_stamp":"2022-02-03T04:12:27.250393+0000","last_clean_scrub_stamp":"2022-02-03T04:12:27.250393+0000","objects_scrubbed":13,"log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:55.183152+0000","stat_sum":{"num_bytes":28373789,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":279,"num_read_kb":308741,"num_write":131,"num_write_kb":35658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":65927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":4,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"180'145","reported_seq":865,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136129+0000","last_change":"2022-02-03T04:13:07.925817+0000","last_active":"2022-02-03T04:13:10.136129+0000","last_peered":"2022-02-03T04:13:10.136129+0000","last_clean":"2022-02-03T04:13:10.136129+0000","last_became_active":"2022-02-03T04:12:42.135593+0000","last_became_peered":"2022-02-03T04:12:42.135593+0000","last_unstale":"2022-02-03T04:13:10.136129+0000","last_undegraded":"2022-02-03T04:13:10.136129+0000","last_fullsized":"2022-02-03T04:13:10.136129+0000","mapping_epoch":202,"log_start":"180'141","ondisk_log_start":"180'141","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'145","last_scrub_stamp":"2022-02-03T04:13:07.925709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:13:07.925709+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.394391+0000","stat_sum":{"num_bytes":14840047,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":186,"num_read_kb":206842,"num_write":62,"num_write_kb":15707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":50867939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"190'83","reported_seq":653,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136121+0000","last_change":"2022-02-03T04:12:42.136823+0000","last_active":"2022-02-03T04:13:10.136121+0000","last_peered":"2022-02-03T04:13:10.136121+0000","last_clean":"2022-02-03T04:13:10.136121+0000","last_became_active":"2022-02-03T04:12:42.136336+0000","last_became_peered":"2022-02-03T04:12:42.136336+0000","last_unstale":"2022-02-03T04:13:10.136121+0000","last_undegraded":"2022-02-03T04:13:10.136121+0000","last_fullsized":"2022-02-03T04:13:10.136121+0000","mapping_epoch":202,"log_start":"171'80","ondisk_log_start":"171'80","created":20,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'83","last_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_deep_scrub":"190'83","last_deep_scrub_stamp":"2022-02-03T04:12:26.146343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:26.146343+0000","objects_scrubbed":3,"log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:42.647507+0000","stat_sum":{"num_bytes":8219415,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":130274,"num_write":43,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":21911059,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"181'279","reported_seq":1097,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.139862+0000","last_change":"2022-02-03T04:12:41.137180+0000","last_active":"2022-02-03T04:13:10.139862+0000","last_peered":"2022-02-03T04:13:10.139862+0000","last_clean":"2022-02-03T04:13:10.139862+0000","last_became_active":"2022-02-03T04:12:41.136735+0000","last_became_peered":"2022-02-03T04:12:41.136735+0000","last_unstale":"2022-02-03T04:13:10.139862+0000","last_undegraded":"2022-02-03T04:13:10.139862+0000","last_fullsized":"2022-02-03T04:13:10.139862+0000","mapping_epoch":201,"log_start":"180'277","ondisk_log_start":"180'277","created":20,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'279","last_scrub_stamp":"2022-02-03T04:12:11.070321+0000","last_deep_scrub":"87'109","last_deep_scrub_stamp":"2022-02-03T04:10:26.906110+0000","last_clean_scrub_stamp":"2022-02-03T04:12:11.070321+0000","objects_scrubbed":11,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:15.311493+0000","stat_sum":{"num_bytes":26457274,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":425,"num_read_kb":535070,"num_write":129,"num_write_kb":32509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":89376022,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":7,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":422,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141638+0000","last_change":"2022-02-03T04:12:38.170890+0000","last_active":"2022-02-03T04:13:10.141638+0000","last_peered":"2022-02-03T04:13:10.141638+0000","last_clean":"2022-02-03T04:13:10.141638+0000","last_became_active":"2022-02-03T04:12:38.170384+0000","last_became_peered":"2022-02-03T04:12:38.170384+0000","last_unstale":"2022-02-03T04:13:10.141638+0000","last_undegraded":"2022-02-03T04:13:10.141638+0000","last_fullsized":"2022-02-03T04:13:10.141638+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2022-02-03T04:12:12.099852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:10:51.035132+0000","last_clean_scrub_stamp":"2022-02-03T04:12:12.099852+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:13.748445+0000","stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"181'298","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.141878+0000","last_change":"2022-02-03T04:12:23.962311+0000","last_active":"2022-02-03T04:13:10.141878+0000","last_peered":"2022-02-03T04:13:10.141878+0000","last_clean":"2022-02-03T04:13:10.141878+0000","last_became_active":"2022-02-03T04:10:59.113356+0000","last_became_peered":"2022-02-03T04:10:59.113356+0000","last_unstale":"2022-02-03T04:13:10.141878+0000","last_undegraded":"2022-02-03T04:13:10.141878+0000","last_fullsized":"2022-02-03T04:13:10.141878+0000","mapping_epoch":116,"log_start":"175'294","ondisk_log_start":"175'294","created":111,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":5,"last_scrub":"181'298","last_scrub_stamp":"2022-02-03T04:12:23.962153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:23.962153+0000","objects_scrubbed":8,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:39.012107+0000","stat_sum":{"num_bytes":17993720,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":101,"num_read_kb":136451,"num_write":36,"num_write_kb":9139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":5,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"190'103","reported_seq":717,"reported_epoch":226,"state":"active+clean","last_fresh":"2022-02-03T04:13:11.148680+0000","last_change":"2022-02-03T04:13:11.148680+0000","last_active":"2022-02-03T04:13:11.148680+0000","last_peered":"2022-02-03T04:13:11.148680+0000","last_clean":"2022-02-03T04:13:11.148680+0000","last_became_active":"2022-02-03T04:13:11.148207+0000","last_became_peered":"2022-02-03T04:13:11.148207+0000","last_unstale":"2022-02-03T04:13:11.148680+0000","last_undegraded":"2022-02-03T04:13:11.148680+0000","last_fullsized":"2022-02-03T04:13:11.148680+0000","mapping_epoch":225,"log_start":"188'101","ondisk_log_start":"188'101","created":111,"last_epoch_clean":226,"parent":"0.0","parent_split_bits":5,"last_scrub":"188'101","last_scrub_stamp":"2022-02-03T04:12:24.966472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:24.966472+0000","objects_scrubbed":3,"log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:34.440143+0000","stat_sum":{"num_bytes":6433973,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4339807,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":2,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"191'204","reported_seq":956,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.136062+0000","last_change":"2022-02-03T04:13:04.736263+0000","last_active":"2022-02-03T04:13:10.136062+0000","last_peered":"2022-02-03T04:13:10.136062+0000","last_clean":"2022-02-03T04:13:10.136062+0000","last_became_active":"2022-02-03T04:13:04.735841+0000","last_became_peered":"2022-02-03T04:13:04.735841+0000","last_unstale":"2022-02-03T04:13:10.136062+0000","last_undegraded":"2022-02-03T04:13:10.136062+0000","last_fullsized":"2022-02-03T04:13:10.136062+0000","mapping_epoch":222,"log_start":"171'199","ondisk_log_start":"171'199","created":111,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":5,"last_scrub":"191'204","last_scrub_stamp":"2022-02-03T04:12:28.692566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:28.692566+0000","objects_scrubbed":5,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:57.056637+0000","stat_sum":{"num_bytes":10499325,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":27222,"num_write":10,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":26308626,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"186'327","reported_seq":1254,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.142057+0000","last_change":"2022-02-03T04:12:41.134599+0000","last_active":"2022-02-03T04:13:10.142057+0000","last_peered":"2022-02-03T04:13:10.142057+0000","last_clean":"2022-02-03T04:13:10.142057+0000","last_became_active":"2022-02-03T04:12:41.134104+0000","last_became_peered":"2022-02-03T04:12:41.134104+0000","last_unstale":"2022-02-03T04:13:10.142057+0000","last_undegraded":"2022-02-03T04:13:10.142057+0000","last_fullsized":"2022-02-03T04:13:10.142057+0000","mapping_epoch":201,"log_start":"186'323","ondisk_log_start":"186'323","created":111,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'327","last_scrub_stamp":"2022-02-03T04:12:17.985205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.985205+0000","objects_scrubbed":16,"log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:47.548621+0000","stat_sum":{"num_bytes":31091584,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":156,"num_read_kb":163033,"num_write":48,"num_write_kb":15182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":81153762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":6,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"180'347","reported_seq":999,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.145122+0000","last_change":"2022-02-03T04:12:17.660361+0000","last_active":"2022-02-03T04:13:10.145122+0000","last_peered":"2022-02-03T04:13:10.145122+0000","last_clean":"2022-02-03T04:13:10.145122+0000","last_became_active":"2022-02-03T04:11:49.126110+0000","last_became_peered":"2022-02-03T04:11:49.126110+0000","last_unstale":"2022-02-03T04:13:10.145122+0000","last_undegraded":"2022-02-03T04:13:10.145122+0000","last_fullsized":"2022-02-03T04:13:10.145122+0000","mapping_epoch":161,"log_start":"180'342","ondisk_log_start":"180'342","created":111,"last_epoch_clean":162,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'347","last_scrub_stamp":"2022-02-03T04:12:17.660263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-02-03T04:09:17.084343+0000","last_clean_scrub_stamp":"2022-02-03T04:12:17.660263+0000","objects_scrubbed":15,"log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:13:28.902662+0000","stat_sum":{"num_bytes":32080619,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":139,"num_read_kb":154239,"num_write":64,"num_write_kb":16178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":18782862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":9,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.16","version":"205'302","reported_seq":1042,"reported_epoch":225,"state":"active+clean","last_fresh":"2022-02-03T04:13:10.137756+0000","last_change":"2022-02-03T04:12:45.157910+0000","last_active":"2022-02-03T04:13:10.137756+0000","last_peered":"2022-02-03T04:13:10.137756+0000","last_clean":"2022-02-03T04:13:10.137756+0000","last_became_active":"2022-02-03T04:11:29.381416+0000","last_became_peered":"2022-02-03T04:11:29.381416+0000","last_unstale":"2022-02-03T04:13:10.137756+0000","last_undegraded":"2022-02-03T04:13:10.137756+0000","last_fullsized":"2022-02-03T04:13:10.137756+0000","mapping_epoch":142,"log_start":"180'295","ondisk_log_start":"180'295","created":111,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"205'302","last_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_deep_scrub":"205'302","last_deep_scrub_stamp":"2022-02-03T04:12:45.157813+0000","last_clean_scrub_stamp":"2022-02-03T04:12:45.157813+0000","objects_scrubbed":7,"log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2022-02-03T04:14:12.020590+0000","stat_sum":{"num_bytes":13542240,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":37,"num_read_kb":44820,"num_write":5,"num_write_kb":1450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":3,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":362454319,"num_objects":169,"num_object_clones":119,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":27,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":7,"num_read":2411,"num_read_kb":2897646,"num_write":897,"num_write_kb":244487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":388,"num_bytes_recovered":627816301,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":77,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":772018176,"data_stored":853278279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":78,"acting":78,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":145419057,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":492,"num_read_kb":29120,"num_write":649,"num_write_kb":253362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":264420763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":276406272,"data_stored":275922081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":12,"acting":12,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137288,"kb_used_data":113088,"kb_used_omap":0,"kb_used_meta":24192,"kb_avail":94234552,"statfs":{"total":96636764160,"available":96496181248,"internally_reserved":0,"allocated":115802112,"data_stored":135333029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24772608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.35199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78700000000000003,"5min":0.874,"15min":0.874},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.55100000000000005,"15min":0.54300000000000004},"min":{"1min":0.377,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":7.3339999999999996,"5min":7.3339999999999996,"15min":7.3339999999999996},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50900000000000001,"15min":0.54100000000000004},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.67700000000000005,"5min":3.4550000000000001,"15min":3.4550000000000001},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.59099999999999997},"min":{"1min":0.40899999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70499999999999996,"5min":3.8919999999999999,"15min":3.8919999999999999},"last":0.497}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44900000000000001,"15min":0.45100000000000001},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58099999999999996,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.46899999999999997,"15min":0.47299999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48699999999999999}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69499999999999995,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.50600000000000001,"15min":0.495},"min":{"1min":0.372,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35799999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.41899999999999998,"15min":0.41399999999999998},"min":{"1min":0.29699999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46400000000000002},"min":{"1min":0.30599999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.46300000000000002}]},{"osd":6,"last update":"Thu Feb 3 04:13:14 2022","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.442,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.49299999999999999},"min":{"1min":0.35699999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.58099999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":166740,"kb_used_data":137820,"kb_used_omap":0,"kb_used_meta":28672,"kb_avail":94205100,"statfs":{"total":96636764160,"available":96466022400,"internally_reserved":0,"allocated":141127680,"data_stored":152810273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29360128},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.50900000000000001,"15min":0.58799999999999997},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66700000000000004,"5min":4.3019999999999996,"15min":4.3019999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71599999999999997,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.59499999999999997,"15min":0.751},"min":{"1min":0.33500000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72499999999999998,"5min":4.5999999999999996,"15min":4.5999999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.495,"15min":0.53100000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.307},"max":{"1min":0.65100000000000002,"5min":2.6429999999999998,"15min":2.6429999999999998},"last":0.67700000000000005}]},{"osd":2,"last update":"Thu Feb 3 04:11:07 2022","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.64100000000000001,"15min":0.67500000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.64000000000000001,"5min":4.383,"15min":4.383},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.624,"15min":0.64700000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.879,"5min":4.0819999999999999,"15min":4.0819999999999999},"last":0.65000000000000002}]},{"osd":3,"last update":"Thu Feb 3 04:12:45 2022","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53300000000000003,"15min":0.56699999999999995},"min":{"1min":0.33900000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.64200000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.497,"5min":0.59299999999999997,"15min":0.64100000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64800000000000002,"5min":4.3490000000000002,"15min":4.3490000000000002},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:24 2022","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.438,"15min":0.42999999999999999},"min":{"1min":0.29499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.627},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54700000000000004,"15min":0.626},"min":{"1min":0.36099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.70199999999999996,"5min":5.3310000000000004,"15min":5.3310000000000004},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52400000000000002,"15min":0.56200000000000006},"min":{"1min":0.32800000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.86799999999999999,"5min":2.746,"15min":2.746},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51300000000000001,"15min":0.51500000000000001},"min":{"1min":0.38100000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79000000000000004,"5min":1.103,"15min":1.103},"last":0.70699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":164620,"kb_used_data":133012,"kb_used_omap":0,"kb_used_meta":31360,"kb_avail":94207220,"statfs":{"total":96636764160,"available":96468193280,"internally_reserved":0,"allocated":136204288,"data_stored":136648473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32112640},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":0.51000000000000001,"15min":0.371},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":22.076000000000001,"5min":22.076000000000001,"15min":22.076000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.35099999999999998,"15min":0.34100000000000003},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":2.1589999999999998},"last":0.42699999999999999}]},{"osd":2,"last update":"Thu Feb 3 04:12:07 2022","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.318,"15min":0.30599999999999999},"min":{"1min":0.19500000000000001,"5min":0.193,"15min":0.193},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.32900000000000001,"15min":0.32800000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.41599999999999998,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:48 2022","interfaces":[{"interface":"back","average":{"1min":0.32200000000000001,"5min":0.34799999999999998,"15min":0.35099999999999998},"min":{"1min":0.24399999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.51100000000000001,"5min":0.878,"15min":0.878},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.36099999999999999,"15min":0.35899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46100000000000002,"5min":1.5429999999999999,"15min":1.5429999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.66700000000000004}]},{"osd":5,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":0.67600000000000005,"15min":0.60499999999999998},"min":{"1min":0.312,"5min":0.309,"15min":0.309},"max":{"1min":12.002000000000001,"5min":12.002000000000001,"15min":12.002000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":0.65900000000000003,"15min":0.59699999999999998},"min":{"1min":0.34300000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":11.265000000000001,"5min":11.265000000000001,"15min":11.265000000000001},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.52000000000000002},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":0.65400000000000003,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":11.972,"5min":11.972,"15min":11.972},"last":0.63700000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:13:07 2022","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":0.66000000000000003,"15min":0.60599999999999998},"min":{"1min":0.38200000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":11.298,"5min":11.298,"15min":11.298},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.58499999999999996,"15min":0.58599999999999997},"min":{"1min":0.41499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.57599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":146864,"kb_used_data":123176,"kb_used_omap":0,"kb_used_meta":23680,"kb_avail":94224976,"statfs":{"total":96636764160,"available":96486375424,"internally_reserved":0,"allocated":126132224,"data_stored":142235603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":24248320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.318,"15min":0.318},"min":{"1min":0.188,"5min":0.186,"15min":0.186},"max":{"1min":0.46800000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.373,"15min":0.34999999999999998},"min":{"1min":0.23599999999999999,"5min":0.218,"15min":0.218},"max":{"1min":0.88500000000000001,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:09 2022","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.379,"15min":0.36499999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.372},{"interface":"front","average":{"1min":0.432,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.38600000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.20399999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.92400000000000004,"5min":1.605,"15min":1.605},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.314,"5min":0.38200000000000001,"15min":0.40699999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.497,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.35999999999999999}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.52600000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55900000000000005,"15min":0.55800000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84399999999999997,"5min":1.105,"15min":1.105},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.98499999999999999,"5min":1.639,"15min":1.639},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.38700000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.78500000000000003,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51500000000000001}]},{"osd":7,"last update":"Thu Feb 3 04:12:12 2022","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.40200000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70499999999999996,"5min":0.88,"15min":0.88},"last":0.441},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.67700000000000005}]}]},{"osd":2,"up_from":195,"seq":837518622731,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87544,"kb_used_data":72560,"kb_used_omap":18,"kb_used_meta":14957,"kb_avail":94284296,"statfs":{"total":96636764160,"available":96547119104,"internally_reserved":0,"allocated":74301440,"data_stored":74704659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19126,"internal_metadata":15316298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]}]},{"osd":3,"up_from":42,"seq":180388626477,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":248144,"kb_used_data":218760,"kb_used_omap":1,"kb_used_meta":29374,"kb_avail":94123696,"statfs":{"total":96636764160,"available":96382664704,"internally_reserved":0,"allocated":224010240,"data_stored":245797411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1080,"internal_metadata":30079944},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.26300000000000001,"5min":0.29299999999999998,"15min":0.29999999999999999},"min":{"1min":0.19800000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.33800000000000002,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.375},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.33500000000000002,"15min":0.33200000000000002},"min":{"1min":0.26900000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.40200000000000002,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34999999999999998,"15min":0.34399999999999997},"min":{"1min":0.221,"5min":0.216,"15min":0.216},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.35399999999999998,"15min":0.34899999999999998},"min":{"1min":0.247,"5min":0.185,"15min":0.185},"max":{"1min":0.64800000000000002,"5min":1.024,"15min":1.024},"last":0.28000000000000003}]},{"osd":2,"last update":"Thu Feb 3 04:11:45 2022","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.26000000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.629,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.34300000000000003,"15min":0.34499999999999997},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.56699999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.36499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.83999999999999997,"5min":1.004,"15min":1.004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.53800000000000003,"15min":0.55000000000000004},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.90800000000000003,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.57199999999999995}]},{"osd":6,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53800000000000003,"15min":0.54800000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.504},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51800000000000002,"15min":0.51900000000000002},"min":{"1min":0.37,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Feb 3 04:12:50 2022","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.53200000000000003,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.749,"5min":1.034,"15min":1.034},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.89000000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.61199999999999999}]}]},{"osd":4,"up_from":134,"seq":575525617689,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":175788,"kb_used_data":153636,"kb_used_omap":37,"kb_used_meta":22106,"kb_avail":94196052,"statfs":{"total":96636764160,"available":96456757248,"internally_reserved":0,"allocated":157323264,"data_stored":158664501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38032,"internal_metadata":22637424},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56399999999999995}]},{"osd":1,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.505},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.52100000000000002}]},{"osd":5,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.60099999999999998}]},{"osd":6,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.54900000000000004}]},{"osd":7,"last update":"Thu Feb 3 04:12:20 2022","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.61499999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":98668,"kb_used_data":82532,"kb_used_omap":0,"kb_used_meta":16128,"kb_avail":94273172,"statfs":{"total":96636764160,"available":96535728128,"internally_reserved":0,"allocated":84512768,"data_stored":88990136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":16515072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44400000000000001,"15min":0.45100000000000001},"min":{"1min":0.253,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60299999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.42799999999999999,"15min":0.41999999999999998},"min":{"1min":0.32600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.379}]},{"osd":1,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45200000000000001,"15min":0.45400000000000001},"min":{"1min":0.33000000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63300000000000001,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.27000000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.58399999999999996}]},{"osd":2,"last update":"Thu Feb 3 04:12:05 2022","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.50900000000000001,"5min":3.121,"15min":3.121},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.441,"5min":0.50800000000000001,"15min":0.498},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.60099999999999998,"5min":3.1509999999999998,"15min":3.1509999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Thu Feb 3 04:12:43 2022","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45800000000000002,"15min":0.46000000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.50800000000000001,"15min":0.51200000000000001},"min":{"1min":0.35799999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Thu Feb 3 04:12:21 2022","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40400000000000003,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.499},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.34399999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56799999999999995}]},{"osd":7,"last update":"Thu Feb 3 04:13:08 2022","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44700000000000001,"15min":0.443},"min":{"1min":0.34899999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.46999999999999997,"15min":0.47799999999999998},"min":{"1min":0.32000000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":21385216,"data_stored":21358053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50597888,"data_stored":50506150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":46559232,"data_stored":46470278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3780,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24190976,"data_stored":24145696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":65728512,"data_stored":65613573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77864960,"data_stored":94686482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":113643520,"data_stored":114740505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":51744768,"data_stored":52797484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1618,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172240896,"data_stored":194742139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":109588480,"data_stored":111644308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59150336,"data_stored":64295318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73691136,"data_stored":86117857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114094080,"data_stored":134254186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2022-02-03T04:13:16.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json , 2022-02-03T04:13:17.180 INFO:teuthology.orchestra.run.smithi090.stdout: 2022-02-03T12:51:21.977 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:13:50.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-02-03T04:13:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: , dumped all 2022-02-03T12:51:21.980 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2022-02-03T04:14:14.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2022-02-03T12:51:21.981 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:14:56.976 INFO:teuthology.misc:Shutting down osd daemons... , 2022-02-03T04:14:56.976 DEBUG:tasks.ceph.osd.0:waiting for process to exit , 2022-02-03T04:14:56.977 INFO:teuthology.orchestra.run:waiting for 300 2022-02-03T12:51:21.982 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:00.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo tar cz -f - -C /var/lib/ceph/crash -- . , 2022-02-03T04:15:00.120 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/remote/smithi160/crash , 2022-02-03T04:15:00.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-02-03T12:51:21.983 WARNING:scrape:Ignoring malparsed backtrace: 2022-02-03T04:15:17.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog , 2022-02-03T04:15:17.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... , 2022-02-03T04:15:17.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-02-03T12:51:43.553 INFO:scrape:Found 11 distinct failure reasons 2022-02-03T12:51:43.554 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi065.front.sepia.ceph.com: ['type=AVC msg=audit(1643867482.856:2980): avc: denied { setgid } for pid=17626 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867419.212:2844): avc: denied { setgid } for pid=15098 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867420.446:2853): avc: denied { setgid } for pid=15121 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867420.242:2849): avc: denied { setgid } for pid=15108 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867481.854:2978): avc: denied { setgid } for pid=17617 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] 2022-02-03T12:51:43.555 INFO:scrape:90 jobs 2022-02-03T12:51:43.556 INFO:scrape:suites intersection: [] 2022-02-03T12:51:43.557 INFO:scrape:suites union: ['0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '1-install/octopus', '1-install/pacific', '1-pg-log-overrides/normal_pg_log', '1-pg-log-overrides/short_pg_log', '2-inject-leak/mon', '2-inject-leak/none', '2-inject-leak/osd', '2-recovery-overrides/{default}', '2-recovery-overrides/{more-async-partial-recovery}', '2-recovery-overrides/{more-async-recovery}', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', '3-scrub-overrides/{max-simultaneous-scrubs-3}', 'backoff/normal', 'backoff/peering', 'backoff/peering_and_degraded', 'centos_latest}', 'ceph', 'cluster/{12-osds', 'clusters/3-mons', 'clusters/9-mons', 'clusters/{fixed-2', 'clusters/{fixed-4', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'crc-failures/default', 'd-balancer/crush-compat', 'd-balancer/on', 'd-thrash/default/{default', 'd-thrash/none', 'debug/mgr', 'fast/fast', 'fast/normal', 'mgr_ttl_cache/disable', 'mgr_ttl_cache/enable', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr-failures/few', 'msgr-failures/many', 'msgr-failures/mon-delay', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'msgr/async-v1only', 'msgr/async-v2only', 'no_pools', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/basic/{ceph', 'rados/dashboard/{0-single-container-host', 'rados/mgr/{clusters/{2-node-mgr}', 'rados/monthrash/{ceph', 'rados/multimon/{clusters/6', 'rados/multimon/{clusters/9', 'rados/objectstore/{backends/filestore-idempotent', 'rados/objectstore/{backends/filestore-idempotent-aio-journal', 'rados/objectstore/{backends/fusestore', 'rados/objectstore/{backends/objectcacher-stress', 'rados/objectstore/{backends/objectstore-filestore-memstore', 'rados/singleton-bluestore/{all/cephtool', 'rados/singleton-nomsgr/{all/cache-fs-trunc', 'rados/singleton-nomsgr/{all/ceph-post-file', 'rados/singleton-nomsgr/{all/health-warnings', 'rados/singleton-nomsgr/{all/lazy_omap_stats_output', 'rados/singleton-nomsgr/{all/librados_hello_world', 'rados/singleton-nomsgr/{all/msgr', 'rados/singleton-nomsgr/{all/osd_stale_reads', 'rados/singleton-nomsgr/{all/pool-access', 'rados/singleton-nomsgr/{all/recovery-unfound-found', 'rados/singleton-nomsgr/{all/version-number-sanity', 'rados/singleton/{all/divergent_priors2', 'rados/singleton/{all/erasure-code-nonregression', 'rados/singleton/{all/max-pg-per-osd.from-primary', 'rados/singleton/{all/osd-recovery', 'rados/singleton/{all/osd-recovery-incomplete', 'rados/singleton/{all/radostool', 'rados/singleton/{all/random-eio', 'rados/singleton/{all/test-noautoscale-flag', 'rados/singleton/{all/thrash-eio', 'rados/singleton/{all/thrash-rados/{thrash-rados', 'rados/singleton/{all/thrash_cache_writeback_proxy_none', 'rados/standalone/{supported-random-distro$/{centos_8}', 'rados/thrash-erasure-code-big/{ceph', 'rados/thrash-erasure-code-isa/{arch/x86_64', 'rados/thrash-erasure-code-overwrites/{bluestore-bitmap', 'rados/thrash-erasure-code-shec/{ceph', 'rados/thrash-erasure-code/{ceph', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'rados/valgrind-leaks/{1-start', 'rados/verify/{centos_latest', 'random-objectstore$/{bluestore-bitmap}', 'random-objectstore$/{bluestore-comp-lz4}', 'random-objectstore$/{bluestore-comp-zlib}', 'random-objectstore$/{bluestore-low-osd-mem-target}', 'random-objectstore$/{bluestore-stupid}', 'recovery-overrides/{default}', 'recovery-overrides/{more-active-recovery}', 'recovery-overrides/{more-async-partial-recovery}', 'recovery-overrides/{more-async-recovery}', 'recovery-overrides/{more-partial-recovery}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'tasks/crash}', 'tasks/dashboard}', 'tasks/e2e}', 'tasks/mon_clock_no_skews}', 'tasks/mon_recovery', 'tasks/mon_recovery}', 'tasks/progress}', 'tasks/prometheus}', 'tasks/rados_api_tests', 'tasks/rados_cls_all', 'tasks/rados_python}', 'tasks/rados_stress_watch}', 'tasks/rados_striper}', 'tasks/rados_workunit_loadgen_mix}', 'thrashers/careful', 'thrashers/default', 'thrashers/fastread', 'thrashers/mapgap', 'thrashers/minsize_recovery', 'thrashers/morepggrow', 'thrashers/none', 'thrashers/one', 'thrashers/pggrow', 'thrashers/sync', 'thrashers/sync-many', 'thrashosds-health', 'thrashosds-health}', 'three-plus-one}', 'validater/lockdep}', 'validater/valgrind}', 'workloads/admin_socket_objecter_requests}', 'workloads/c2c}', 'workloads/cache}', 'workloads/dedup-io-snaps}', 'workloads/ec-pool-snaps-few-objects-overwrites}', 'workloads/ec-rados-plugin=isa-k=2-m=1}', 'workloads/ec-rados-plugin=jerasure-k=4-m=2}', 'workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}', 'workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'workloads/ec-small-objects-fast-read-overwrites}', 'workloads/ec-small-objects-overwrites}', 'workloads/ec-small-objects}', 'workloads/pool-snaps-few-objects}', 'workloads/rados_5925}', 'workloads/rados_api_tests}', 'workloads/rados_mon_osdmap_prune}', 'workloads/rados_mon_workunits}', 'workloads/radosbench-high-concurrency}', 'workloads/radosbench}', 'workloads/redirect_set_object}', 'workloads/redirect}', 'workloads/small-objects-balanced}', 'workloads/snaps-few-objects-balanced}', 'workloads/snaps-few-objects}', 'workloads/test_rbd_api}'] 2022-02-03T12:51:43.558 INFO:scrape: 2022-02-03T12:51:43.558 INFO:scrape:Failure: Command failed on smithi104 with status 1: 'sudo yum install -y kernel' 2022-02-03T12:51:43.559 INFO:scrape:16 jobs: ['6657418', '6657372', '6657431', '6657491', '6657269', '6657335', '6657446', '6657465', '6657392', '6657281', '6657328', '6657207', '6657406', '6657312', '6657191', '6657248'] 2022-02-03T12:51:43.559 INFO:scrape:suites intersection: [] 2022-02-03T12:51:43.560 INFO:scrape:suites union: ['0-nvme-loop', '0-size-min-size-overrides/2-size-2-min-size', '0-size-min-size-overrides/3-size-2-min-size', '0-start', '1-install/nautilus-v1only', '1-install/pacific', '1-start', '1-tasks', '2-ops/repave-all}', '2-ops/rm-zap-add}', '2-ops/rm-zap-flag}', '2-ops/rm-zap-wait}', '2-ops/rmdir-reactivate}', '2-services/basic', '3-final}', 'agent/off', 'agent/on', 'backoff/normal', 'backoff/peering_and_degraded', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'd-balancer/on', 'fixed-2', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/none', 'msgr-failures/osd-delay', 'msgr/async-v2only', 'objectstore/bluestore-hybrid', 'rados', 'rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0', 'rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.2_container_tools_3.0}', 'rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0', 'rados/singleton/{all/lost-unfound-delete', 'rados/thrash-old-clients/{0-distro$/{centos_8.3_container_tools_3.0}', 'rados/upgrade/parallel/{0-random-distro$/{centos_8.3_container_tools_3.0}', 'rados_api', 'rados_loadgenbig', 'rbd_import_export', 'start}', 'supported-random-distro$/{ubuntu_latest}}', 'test_rbd_api', 'test_rbd_python}}', 'thrashers/careful', 'thrashers/default', 'thrashosds-health', 'three-plus-one}', 'upgrade-sequence', 'workload/{ec-rados-default', 'workloads/cache-snaps}', 'workloads/radosbench}'] 2022-02-03T12:51:43.560 INFO:scrape: 2022-02-03T12:51:43.561 INFO:scrape:Failure: Command failed on smithi204 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2fae4342-84ac-11ec-8c35-001a4aab830c -- ceph mon dump -f json' 2022-02-03T12:51:43.561 INFO:scrape:1 jobs: ['6657265'] 2022-02-03T12:51:43.562 INFO:scrape:suites: ['0-nvme-loop', '1-start', '2-ops/rm-zap-wait}', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0'] 2022-02-03T12:51:43.562 INFO:scrape: 2022-02-03T12:51:43.563 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi125.front.sepia.ceph.com: ['type=AVC msg=audit(1643866443.420:2853): avc: denied { setgid } for pid=15125 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866498.127:2978): avc: denied { setgid } for pid=17623 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866442.488:2848): avc: denied { setgid } for pid=15115 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866499.055:2983): avc: denied { setgid } for pid=17634 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] 2022-02-03T12:51:43.563 INFO:scrape:4 jobs: ['6657346', '6657237', '6657243', '6657420'] 2022-02-03T12:51:43.564 INFO:scrape:suites intersection: ['rados'] 2022-02-03T12:51:43.564 INFO:scrape:suites union: ['0-size-min-size-overrides/3-size-2-min-size', '1-install/nautilus', '1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-partial-recovery}', '3-scrub-overrides/{default}', 'backoff/peering_and_degraded', 'ceph', 'clusters/{fixed-2', 'clusters/{openstack', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'd-balancer/on', 'mon_election/classic', 'mon_election/connectivity', 'msgr-failures/few', 'msgr-failures/osd-dispatch-delay', 'msgr/async', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'openstack}', 'rados', 'rados/singleton-nomsgr/{all/admin_socket_output', 'rados/singleton/{all/pg-removal-interruption', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_8}}', 'thrashers/mapgap', 'thrashers/morepggrow', 'thrashosds-health', 'three-plus-one}', 'workloads/cache-pool-snaps-readproxy}', 'workloads/snaps-few-objects}'] 2022-02-03T12:51:43.565 INFO:scrape: 2022-02-03T12:51:43.565 INFO:scrape:Failure: Command failed on smithi005 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45222710-84ad-11ec-8c35-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nHOST=$(hostname -s)\nOSD=$(ceph orch ps $HOST | grep osd | head -n 1 | awk \'"\'"\'{print $1}\'"\'"\')\necho "host $HOST, osd $OSD"\nceph orch daemon stop $OSD\nwhile ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\nceph auth export $OSD > k\nceph orch daemon rm $OSD --force\nceph orch ps --refresh\nwhile ceph orch ps | grep $OSD ; do sleep 5 ; done\nceph auth add $OSD -i k\nceph cephadm osd activate $HOST\nwhile ! ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\n\'' 2022-02-03T12:51:43.566 INFO:scrape:2 jobs: ['6657284', '6657367'] 2022-02-03T12:51:43.566 INFO:scrape:suites intersection: ['0-nvme-loop', '1-start', '2-ops/rmdir-reactivate}'] 2022-02-03T12:51:43.567 INFO:scrape:suites union: ['0-nvme-loop', '1-start', '2-ops/rmdir-reactivate}', 'rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun', 'rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8'] 2022-02-03T12:51:43.567 INFO:scrape: 2022-02-03T12:51:43.568 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' 2022-02-03T12:51:43.568 INFO:scrape:1 jobs: ['6657187'] 2022-02-03T12:51:43.569 INFO:scrape:suites: ['1-pg-log-overrides/short_pg_log', '2-recovery-overrides/{more-async-partial-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/default', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/fastclose', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'supported-random-distro$/{centos_8}', 'thrashers/default', 'thrashosds-health', 'workloads/dedup-io-snaps}'] 2022-02-03T12:51:43.569 INFO:scrape: 2022-02-03T12:51:43.569 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/rados/test.sh 2022-02-03T12:51:43.570 INFO:scrape:1 jobs: ['6657473'] 2022-02-03T12:51:43.570 INFO:scrape:suites: ['ceph', 'clusters/{fixed-2', 'd-thrash/default/{default', 'mon_election/connectivity', 'msgr-failures/few', 'msgr/async-v2only', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rados', 'rados/verify/{centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}', 'validater/valgrind}'] 2022-02-03T12:51:43.571 INFO:scrape: 2022-02-03T12:51:43.572 INFO:scrape:Failure: Command failed (workunit test cephadm/test_cephadm.sh) on smithi006 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' 2022-02-03T12:51:43.572 INFO:scrape:2 jobs: ['6657398', '6657275'] 2022-02-03T12:51:43.572 INFO:scrape:suites intersection: ['task/test_cephadm}'] 2022-02-03T12:51:43.573 INFO:scrape:suites union: ['mon_election/classic', 'mon_election/connectivity', 'rados/cephadm/workunits/{agent/off', 'rados/cephadm/workunits/{agent/on', 'task/test_cephadm}'] 2022-02-03T12:51:43.573 INFO:scrape: 2022-02-03T12:51:43.574 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2022-02-03T12:51:43.574 INFO:scrape:1 jobs: ['6657413'] 2022-02-03T12:51:43.575 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{more-active-recovery}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/peering', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{rhel_8}', 'thrashers/none', 'thrashosds-health', 'workloads/rados_api_tests}'] 2022-02-03T12:51:43.575 INFO:scrape: 2022-02-03T12:51:43.576 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1643867339.108:2937): avc: denied { setgid } for pid=16490 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867340.060:2942): avc: denied { setgid } for pid=16500 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] 2022-02-03T12:51:43.576 INFO:scrape:1 jobs: ['6657371'] 2022-02-03T12:51:43.577 INFO:scrape:suites: ['0-size-min-size-overrides/2-size-2-min-size', '1-install/nautilus-v2only', 'backoff/peering', 'ceph', 'clusters/{openstack', 'd-balancer/crush-compat', 'mon_election/classic', 'msgr-failures/fastclose', 'rados', 'rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools}', 'thrashers/mapgap', 'thrashosds-health', 'three-plus-one}', 'workloads/rbd_cls}'] 2022-02-03T12:51:43.577 INFO:scrape: 2022-02-03T12:51:43.578 INFO:scrape:Failure: Test failure: test_update_export_with_invalid_values (tasks.cephfs.test_nfs.TestNFS) 2022-02-03T12:51:43.578 INFO:scrape:1 jobs: ['6657199'] 2022-02-03T12:51:43.579 INFO:scrape:suites: ['mon_election/classic', 'rados/cephadm/workunits/{agent/on', 'task/test_nfs}'] 2022-02-03T12:51:43.579 INFO:scrape: 2022-02-03T12:51:43.593 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/ failed: 120 dead: 0 running: 0 waiting: 0 queued: 0 passed: 207 Fail ================================================================= [6657185] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657185/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657185/ SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC msg=audit(1643861071.055:2850): avc: denied { setgid } for pid=15124 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861072.100:2856): avc: denied { setgid } for pid=15134 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861071.841:2854): avc: denied { setgid } for pid=15130 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861130.714:2983): avc: denied { setgid } for pid=17658 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861129.446:2981): avc: denied { setgid } for pid=17649 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657186] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:27:13 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657186/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657186/ SELinux denials found on ubuntu@smithi066.front.sepia.ceph.com: ['type=AVC msg=audit(1643861215.042:3027): avc: denied { setgid } for pid=18247 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861158.417:2896): avc: denied { setgid } for pid=16361 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861214.006:3022): avc: denied { setgid } for pid=18236 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861158.211:2892): avc: denied { setgid } for pid=16352 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861157.234:2887): avc: denied { setgid } for pid=16342 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657187] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:12:25 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657187/ Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max- seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0' [6657189] rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:13:07 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657189/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657189/ SELinux denials found on ubuntu@smithi018.front.sepia.ceph.com: ['type=AVC msg=audit(1643861162.372:2981): avc: denied { setgid } for pid=17657 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861100.939:2846): avc: denied { setgid } for pid=15129 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861161.345:2976): avc: denied { setgid } for pid=17646 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861101.993:2848): avc: denied { setgid } for pid=15137 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657191] rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657191/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afbe2556db6b4806a90fef5836054e74 Command failed on smithi107 with status 1: 'sudo yum install -y kernel' [6657193] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} ----------------------------------------------------------------- time: 00:18:46 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657193/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657193/ SELinux denials found on ubuntu@smithi074.front.sepia.ceph.com: ['type=AVC msg=audit(1643861260.599:2858): avc: denied { setgid } for pid=15167 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861260.846:2860): avc: denied { setgid } for pid=15171 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861328.984:2988): avc: denied { setgid } for pid=17685 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861329.632:2992): avc: denied { setgid } for pid=17691 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861329.886:2994): avc: denied { setgid } for pid=17696 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861259.935:2854): avc: denied { setgid } for pid=15161 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657194] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} ----------------------------------------------------------------- time: 00:24:20 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657194/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657194/ SELinux denials found on ubuntu@smithi123.front.sepia.ceph.com: ['type=AVC msg=audit(1643861261.052:2842): avc: denied { setgid } for pid=15087 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861262.077:2847): avc: denied { setgid } for pid=15097 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861321.520:2975): avc: denied { setgid } for pid=17603 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861322.178:2979): avc: denied { setgid } for pid=17609 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861322.456:2981): avc: denied { setgid } for pid=17614 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657197] rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:43:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657197/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657197/ SELinux denials found on ubuntu@smithi096.front.sepia.ceph.com: ['type=AVC msg=audit(1643861369.798:2846): avc: denied { setgid } for pid=15071 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861430.580:2986): avc: denied { setgid } for pid=17600 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861370.698:2848): avc: denied { setgid } for pid=15079 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861429.603:2981): avc: denied { setgid } for pid=17589 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657199] rados/cephadm/workunits/{agent/on mon_election/classic task/test_nfs} ----------------------------------------------------------------- time: 00:29:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657199/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3084b5bdd8fd410bb93be9486cb5a573 Test failure: test_update_export_with_invalid_values (tasks.cephfs.test_nfs.TestNFS) [6657200] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:24:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657200/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657200/ SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1643861626.071:2987): avc: denied { setgid } for pid=17632 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861570.263:2848): avc: denied { setgid } for pid=15113 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861571.119:2853): avc: denied { setgid } for pid=15123 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861625.758:2985): avc: denied { setgid } for pid=17627 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861625.113:2981): avc: denied { setgid } for pid=17621 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657201] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:16:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657201/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657201/ SELinux denials found on ubuntu@smithi131.front.sepia.ceph.com: ['type=AVC msg=audit(1643861542.873:2848): avc: denied { setgid } for pid=15132 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861600.340:2985): avc: denied { setgid } for pid=17654 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861601.124:2987): avc: denied { setgid } for pid=17671 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861599.612:2981): avc: denied { setgid } for pid=17640 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861543.838:2853): avc: denied { setgid } for pid=15142 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657203] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657203/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657203/ SELinux denials found on ubuntu@smithi087.front.sepia.ceph.com: ['type=AVC msg=audit(1643861600.156:2855): avc: denied { setgid } for pid=15106 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861657.958:2990): avc: denied { setgid } for pid=17635 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861601.159:2860): avc: denied { setgid } for pid=15116 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861656.939:2985): avc: denied { setgid } for pid=17624 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657206] rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:22:02 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657206/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657206/ SELinux denials found on ubuntu@smithi124.front.sepia.ceph.com: ['type=AVC msg=audit(1643861752.210:2976): avc: denied { setgid } for pid=17616 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861697.518:2846): avc: denied { setgid } for pid=15100 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861753.162:2978): avc: denied { setgid } for pid=17625 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861698.415:2848): avc: denied { setgid } for pid=15116 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657207] rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657207/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657207/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d99cdc9d363442b680181c40bdd5559a Command failed on smithi173 with status 1: 'sudo yum install -y kernel' [6657208] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} ----------------------------------------------------------------- time: 00:21:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657208/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657208/ SELinux denials found on ubuntu@smithi073.front.sepia.ceph.com: ['type=AVC msg=audit(1643861769.386:2976): avc: denied { setgid } for pid=17578 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861710.275:2846): avc: denied { setgid } for pid=15063 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861709.321:2841): avc: denied { setgid } for pid=15053 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861768.392:2974): avc: denied { setgid } for pid=17569 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657209] rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:27:55 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657209/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657209/ SELinux denials found on ubuntu@smithi189.front.sepia.ceph.com: ['type=AVC msg=audit(1643861786.646:2846): avc: denied { setgid } for pid=15076 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861844.042:2981): avc: denied { setgid } for pid=17603 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861843.043:2976): avc: denied { setgid } for pid=17592 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861787.550:2851): avc: denied { setgid } for pid=15086 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657210] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_mon_osdmap_prune} ----------------------------------------------------------------- time: 00:40:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657210/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657210/ SELinux denials found on ubuntu@smithi083.front.sepia.ceph.com: ['type=AVC msg=audit(1643861770.028:2847): avc: denied { setgid } for pid=15067 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861769.017:2845): avc: denied { setgid } for pid=15059 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861828.690:2980): avc: denied { setgid } for pid=17586 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861827.747:2975): avc: denied { setgid } for pid=17575 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657212] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:26:20 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657212/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657212/ SELinux denials found on ubuntu@smithi045.front.sepia.ceph.com: ['type=AVC msg=audit(1643861872.757:2858): avc: denied { setgid } for pid=15109 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861930.799:2987): avc: denied { setgid } for pid=17634 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861871.758:2853): avc: denied { setgid } for pid=15099 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861931.142:2989): avc: denied { setgid } for pid=17641 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861929.518:2983): avc: denied { setgid } for pid=17618 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657213] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} ----------------------------------------------------------------- time: 00:22:40 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657213/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657213/ SELinux denials found on ubuntu@smithi176.front.sepia.ceph.com: ['type=AVC msg=audit(1643861872.188:2845): avc: denied { setgid } for pid=15121 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861933.952:2977): avc: denied { setgid } for pid=17646 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861933.034:2975): avc: denied { setgid } for pid=17637 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861873.212:2850): avc: denied { setgid } for pid=15131 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657217] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:02 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657217/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657217/ SELinux denials found on ubuntu@smithi026.front.sepia.ceph.com: ['type=AVC msg=audit(1643861951.771:2855): avc: denied { setgid } for pid=15144 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861950.844:2850): avc: denied { setgid } for pid=15134 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862015.444:2987): avc: denied { setgid } for pid=17669 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862014.357:2982): avc: denied { setgid } for pid=17656 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657219] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:31:39 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657219/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657219/ SELinux denials found on ubuntu@smithi084.front.sepia.ceph.com: ['type=AVC msg=audit(1643862029.260:2992): avc: denied { setgid } for pid=17677 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861969.897:2858): avc: denied { setgid } for pid=15149 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861969.101:2854): avc: denied { setgid } for pid=15143 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862028.341:2990): avc: denied { setgid } for pid=17668 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643861970.165:2860): avc: denied { setgid } for pid=15153 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657227] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:15:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657227/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657227/ SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC msg=audit(1643862207.871:3029): avc: denied { setgid } for pid=18299 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862208.857:3034): avc: denied { setgid } for pid=18310 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862147.172:2898): avc: denied { setgid } for pid=16405 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862148.304:2903): avc: denied { setgid } for pid=16421 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657231] rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:09:31 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657231/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657231/ SELinux denials found on ubuntu@smithi141.front.sepia.ceph.com: ['type=AVC msg=audit(1643862190.031:2977): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862122.058:2847): avc: denied { setgid } for pid=15134 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862123.096:2852): avc: denied { setgid } for pid=15144 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862191.360:2982): avc: denied { setgid } for pid=17671 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657237] rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:06:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657237/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657237/ SELinux denials found on ubuntu@smithi204.front.sepia.ceph.com: ['type=AVC msg=audit(1643862318.926:2846): avc: denied { setgid } for pid=15093 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862377.290:2978): avc: denied { setgid } for pid=17617 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862320.058:2851): avc: denied { setgid } for pid=15109 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862375.730:2976): avc: denied { setgid } for pid=17608 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657238] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/dashboard} ----------------------------------------------------------------- time: 01:11:29 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657238/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657238/ SELinux denials found on ubuntu@smithi068.front.sepia.ceph.com: ['type=AVC msg=audit(1643862416.461:2902): avc: denied { setgid } for pid=16381 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862415.447:2900): avc: denied { setgid } for pid=16373 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862481.093:3030): avc: denied { setgid } for pid=18262 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862482.103:3032): avc: denied { setgid } for pid=18271 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657243] rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:15:48 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657243/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657243/ SELinux denials found on ubuntu@smithi196.front.sepia.ceph.com: ['type=AVC msg=audit(1643862496.898:2978): avc: denied { setgid } for pid=17632 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862497.180:2980): avc: denied { setgid } for pid=17637 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862435.909:2841): avc: denied { setgid } for pid=15110 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862436.935:2846): avc: denied { setgid } for pid=15120 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862496.269:2974): avc: denied { setgid } for pid=17626 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657244] rados/standalone/{supported-random-distro$/{centos_8} workloads/c2c} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657244/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657244/ SELinux denials found on ubuntu@smithi154.front.sepia.ceph.com: ['type=AVC msg=audit(1643862502.307:2862): avc: denied { setgid } for pid=15961 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862475.557:2856): avc: denied { setgid } for pid=15179 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862474.538:2851): avc: denied { setgid } for pid=15169 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862503.209:2864): avc: denied { setgid } for pid=15970 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657246] rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657246/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657246/ SELinux denials found on ubuntu@smithi145.front.sepia.ceph.com: ['type=AVC msg=audit(1643862519.124:2975): avc: denied { setgid } for pid=17634 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862520.085:2980): avc: denied { setgid } for pid=17645 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862460.507:2845): avc: denied { setgid } for pid=15118 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862461.513:2850): avc: denied { setgid } for pid=15128 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657248] rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657248/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657248/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8de919110a3a4a3884df6bc06c0d0b9c Command failed on smithi121 with status 1: 'sudo yum install -y kernel' [6657250] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:21:30 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657250/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657250/ SELinux denials found on ubuntu@smithi097.front.sepia.ceph.com: ['type=AVC msg=audit(1643862677.931:2859): avc: denied { setgid } for pid=15129 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862676.737:2850): avc: denied { setgid } for pid=15102 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862741.771:2989): avc: denied { setgid } for pid=17640 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862677.746:2855): avc: denied { setgid } for pid=15120 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862740.904:2984): avc: denied { setgid } for pid=17629 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657251] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657251/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657251/ SELinux denials found on ubuntu@smithi074.front.sepia.ceph.com: ['type=AVC msg=audit(1643862834.422:2973): avc: denied { setgid } for pid=17633 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862776.773:2848): avc: denied { setgid } for pid=15126 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862835.399:2978): avc: denied { setgid } for pid=17644 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862775.823:2843): avc: denied { setgid } for pid=15116 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657252] rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:20:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657252/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657252/ SELinux denials found on ubuntu@smithi107.front.sepia.ceph.com: ['type=AVC msg=audit(1643862751.085:2848): avc: denied { setgid } for pid=15069 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862752.032:2853): avc: denied { setgid } for pid=15079 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862819.941:2983): avc: denied { setgid } for pid=17596 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643862818.989:2978): avc: denied { setgid } for pid=17585 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657259] rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:47 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657259/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657259/ SELinux denials found on ubuntu@smithi061.front.sepia.ceph.com: ['type=AVC msg=audit(1643863029.158:2875): avc: denied { setgid } for pid=15145 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863087.417:3003): avc: denied { setgid } for pid=17655 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863028.085:2869): avc: denied { setgid } for pid=15135 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863028.834:2873): avc: denied { setgid } for pid=15141 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863088.486:3008): avc: denied { setgid } for pid=17668 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657261] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} ----------------------------------------------------------------- time: 00:30:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657261/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657261/ SELinux denials found on ubuntu@smithi167.front.sepia.ceph.com: ['type=AVC msg=audit(1643863133.521:3003): avc: denied { setgid } for pid=17630 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863132.568:2998): avc: denied { setgid } for pid=17619 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863075.873:2865): avc: denied { setgid } for pid=15103 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863076.877:2870): avc: denied { setgid } for pid=15113 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657263] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657263/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657263/ SELinux denials found on ubuntu@smithi066.front.sepia.ceph.com: ['type=AVC msg=audit(1643863251.230:2983): avc: denied { setgid } for pid=17615 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863194.266:2854): avc: denied { setgid } for pid=15104 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863250.153:2981): avc: denied { setgid } for pid=17606 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863193.050:2845): avc: denied { setgid } for pid=15085 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863194.066:2850): avc: denied { setgid } for pid=15095 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657264] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_python} ----------------------------------------------------------------- time: 00:12:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657264/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657264/ SELinux denials found on ubuntu@smithi006.front.sepia.ceph.com: ['type=AVC msg=audit(1643863215.742:2875): avc: denied { setgid } for pid=15986 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863216.736:2880): avc: denied { setgid } for pid=15997 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863188.115:2864): avc: denied { setgid } for pid=15201 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863187.148:2859): avc: denied { setgid } for pid=15191 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657265] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:13:58 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657265/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63aee174586f4ffd8805741308b79fef Command failed on smithi204 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2fae4342-84ac-11ec-8c35-001a4aab830c -- ceph mon dump -f json' [6657266] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657266/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657266/ SELinux denials found on ubuntu@smithi139.front.sepia.ceph.com: ['type=AVC msg=audit(1643863165.707:2858): avc: denied { setgid } for pid=15128 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863164.743:2853): avc: denied { setgid } for pid=15118 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863225.240:2990): avc: denied { setgid } for pid=17648 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863224.313:2985): avc: denied { setgid } for pid=17637 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657269] rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657269/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657269/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ec401407d444067972a9a9c0aa42423 Command failed on smithi173 with status 1: 'sudo yum install -y kernel' [6657275] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_cephadm} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657275/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657275/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95b22882d4724c979944b99814eff31c Command failed (workunit test cephadm/test_cephadm.sh) on smithi073 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6657281] rados/thrash-old-clients/{0-distro$/{centos_8.3_container_tools_3.0} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657281/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657281/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42068d6fa3c44b0da743f5cb8fa18e78 Command failed on smithi154 with status 1: 'sudo yum install -y kernel' [6657284] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657284/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657284/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdf778ddbc8e495181f112ef17ecf1c1 Command failed on smithi005 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 45222710-84ad-11ec-8c35-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nHOST=$(hostname -s)\nOSD=$(ceph orch ps $HOST | grep osd | head -n 1 | awk \'"\'"\'{print $1}\'"\'"\')\necho "host $HOST, osd $OSD"\nceph orch daemon stop $OSD\nwhile ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\nceph auth export $OSD > k\nceph orch daemon rm $OSD --force\nceph orch ps --refresh\nwhile ceph orch ps | grep $OSD ; do sleep 5 ; done\nceph auth add $OSD -i k\nceph cephadm osd activate $HOST\nwhile ! ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\n\'' [6657290] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/rados_stress_watch} ----------------------------------------------------------------- time: 00:13:15 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657290/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657290/ SELinux denials found on ubuntu@smithi074.front.sepia.ceph.com: ['type=AVC msg=audit(1643863710.248:2855): avc: denied { setgid } for pid=15183 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863738.193:2865): avc: denied { setgid } for pid=15982 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863709.608:2851): avc: denied { setgid } for pid=15177 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863737.299:2863): avc: denied { setgid } for pid=15973 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863710.529:2857): avc: denied { setgid } for pid=15187 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657297] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:24:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657297/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657297/ SELinux denials found on ubuntu@smithi185.front.sepia.ceph.com: ['type=AVC msg=audit(1643863888.073:2851): avc: denied { setgid } for pid=15143 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863956.916:2976): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863887.066:2846): avc: denied { setgid } for pid=15133 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863957.870:2981): avc: denied { setgid } for pid=17661 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657298] rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:07:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657298/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657298/ SELinux denials found on ubuntu@smithi064.front.sepia.ceph.com: ['type=AVC msg=audit(1643863851.993:2848): avc: denied { setgid } for pid=15120 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863916.261:2983): avc: denied { setgid } for pid=17646 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863853.108:2853): avc: denied { setgid } for pid=15131 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863915.268:2978): avc: denied { setgid } for pid=17635 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657301] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:25:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657301/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657301/ SELinux denials found on ubuntu@smithi124.front.sepia.ceph.com: ['type=AVC msg=audit(1643863892.429:2850): avc: denied { setgid } for pid=15130 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863893.377:2855): avc: denied { setgid } for pid=15140 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863947.671:2980): avc: denied { setgid } for pid=17646 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863948.640:2985): avc: denied { setgid } for pid=17657 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643863948.839:2989): avc: denied { setgid } for pid=17670 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657307] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:14:44 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657307/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657307/ SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1643864196.599:2853): avc: denied { setgid } for pid=15139 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864269.027:2983): avc: denied { setgid } for pid=17659 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864269.954:2985): avc: denied { setgid } for pid=17668 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864197.596:2855): avc: denied { setgid } for pid=15147 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657312] rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657312/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=beb33b39ab4949488998e9aacddd2443 Command failed on smithi107 with status 1: 'sudo yum install -y kernel' [6657313] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_striper} ----------------------------------------------------------------- time: 00:06:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657313/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657313/ SELinux denials found on ubuntu@smithi123.front.sepia.ceph.com: ['type=AVC msg=audit(1643864442.002:2856): avc: denied { setgid } for pid=15124 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864470.423:2871): avc: denied { setgid } for pid=15927 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864469.696:2867): avc: denied { setgid } for pid=15921 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864442.904:2858): avc: denied { setgid } for pid=15132 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864470.723:2873): avc: denied { setgid } for pid=15933 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657315] rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:08:15 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657315/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657315/ SELinux denials found on ubuntu@smithi065.front.sepia.ceph.com: ['type=AVC msg=audit(1643864469.262:2852): avc: denied { setgid } for pid=15129 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864528.500:2979): avc: denied { setgid } for pid=17648 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864468.261:2847): avc: denied { setgid } for pid=15119 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864527.491:2977): avc: denied { setgid } for pid=17639 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657320] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:41:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657320/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657320/ SELinux denials found on ubuntu@smithi060.front.sepia.ceph.com: ['type=AVC msg=audit(1643864606.636:2989): avc: denied { setgid } for pid=17641 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864547.217:2856): avc: denied { setgid } for pid=15119 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864607.646:2994): avc: denied { setgid } for pid=17652 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864548.249:2861): avc: denied { setgid } for pid=15129 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657322] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/e2e} ----------------------------------------------------------------- time: 00:18:10 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657322/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657322/ SELinux denials found on ubuntu@smithi050.front.sepia.ceph.com: ['type=AVC msg=audit(1643864637.129:3043): avc: denied { setgid } for pid=18317 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864638.156:3048): avc: denied { setgid } for pid=18328 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864577.028:2910): avc: denied { setgid } for pid=16426 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864577.992:2915): avc: denied { setgid } for pid=16436 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657326] rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:26:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657326/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657326/ SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1643864722.129:2867): avc: denied { setgid } for pid=15135 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864786.119:3000): avc: denied { setgid } for pid=17655 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864721.170:2865): avc: denied { setgid } for pid=15127 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864785.159:2995): avc: denied { setgid } for pid=17644 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657328] rados/thrash-old-clients/{0-distro$/{centos_8.3_container_tools_3.0} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v1only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657328/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0780d9f26a134af4b356ad662fe1ab65 Command failed on smithi122 with status 1: 'sudo yum install -y kernel' [6657332] rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 02:24:42 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657332/ SELinux denials found on ubuntu@smithi107.front.sepia.ceph.com: ['type=AVC msg=audit(1643864875.981:2975): avc: denied { setgid } for pid=17625 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864875.027:2970): avc: denied { setgid } for pid=17614 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864818.624:2840): avc: denied { setgid } for pid=15098 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864819.562:2842): avc: denied { setgid } for pid=15106 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657333] rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:02 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657333/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657333/ SELinux denials found on ubuntu@smithi099.front.sepia.ceph.com: ['type=AVC msg=audit(1643864852.527:2849): avc: denied { setgid } for pid=15112 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864907.378:2980): avc: denied { setgid } for pid=17632 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864908.272:2982): avc: denied { setgid } for pid=17641 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864853.785:2855): avc: denied { setgid } for pid=15132 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864853.533:2853): avc: denied { setgid } for pid=15118 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657334] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:08:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657334/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657334/ SELinux denials found on ubuntu@smithi039.front.sepia.ceph.com: ['type=AVC msg=audit(1643864916.194:2846): avc: denied { setgid } for pid=15116 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864985.092:2982): avc: denied { setgid } for pid=17647 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864984.148:2976): avc: denied { setgid } for pid=17636 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864917.192:2848): avc: denied { setgid } for pid=15124 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643864984.830:2980): avc: denied { setgid } for pid=17642 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657335] rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657335/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657335/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77ee2a6b56274755917eaee76a5994cb Command failed on smithi078 with status 1: 'sudo yum install -y kernel' [6657339] rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657339/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657339/ SELinux denials found on ubuntu@smithi083.front.sepia.ceph.com: ['type=AVC msg=audit(1643866300.999:2847): avc: denied { setgid } for pid=15087 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866301.873:2849): avc: denied { setgid } for pid=15095 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866357.758:2977): avc: denied { setgid } for pid=17595 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866358.425:2981): avc: denied { setgid } for pid=17609 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866358.719:2983): avc: denied { setgid } for pid=17614 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657340] rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:10:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657340/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657340/ SELinux denials found on ubuntu@smithi066.front.sepia.ceph.com: ['type=AVC msg=audit(1643866456.468:2986): avc: denied { setgid } for pid=17636 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866455.031:2980): avc: denied { setgid } for pid=17615 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866456.165:2984): avc: denied { setgid } for pid=17621 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866398.563:2849): avc: denied { setgid } for pid=15107 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866397.405:2847): avc: denied { setgid } for pid=15099 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657342] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} ----------------------------------------------------------------- time: 00:26:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657342/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657342/ SELinux denials found on ubuntu@smithi061.front.sepia.ceph.com: ['type=AVC msg=audit(1643866364.942:2858): avc: denied { setgid } for pid=15145 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866363.880:2856): avc: denied { setgid } for pid=15137 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866423.781:2988): avc: denied { setgid } for pid=17665 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866422.328:2986): avc: denied { setgid } for pid=17656 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657346] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} ----------------------------------------------------------------- time: 00:24:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657346/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657346/ SELinux denials found on ubuntu@smithi125.front.sepia.ceph.com: ['type=AVC msg=audit(1643866443.420:2853): avc: denied { setgid } for pid=15125 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866498.127:2978): avc: denied { setgid } for pid=17623 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866442.488:2848): avc: denied { setgid } for pid=15115 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866499.055:2983): avc: denied { setgid } for pid=17634 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657350] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:30:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657350/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657350/ SELinux denials found on ubuntu@smithi074.front.sepia.ceph.com: ['type=AVC msg=audit(1643866683.823:2974): avc: denied { setgid } for pid=17640 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866682.846:2972): avc: denied { setgid } for pid=17631 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866578.409:2847): avc: denied { setgid } for pid=15114 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866577.451:2842): avc: denied { setgid } for pid=15104 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657351] rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:12:31 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657351/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657351/ SELinux denials found on ubuntu@smithi152.front.sepia.ceph.com: ['type=AVC msg=audit(1643866652.777:2977): avc: denied { setgid } for pid=17641 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866591.779:2849): avc: denied { setgid } for pid=15133 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866653.712:2982): avc: denied { setgid } for pid=17652 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866590.752:2847): avc: denied { setgid } for pid=15125 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657353] rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} ----------------------------------------------------------------- time: 00:16:52 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657353/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657353/ SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1643866681.143:2846): avc: denied { setgid } for pid=15101 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866680.499:2842): avc: denied { setgid } for pid=15095 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866681.456:2848): avc: denied { setgid } for pid=15105 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866739.149:2976): avc: denied { setgid } for pid=17615 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643866739.746:2978): avc: denied { setgid } for pid=17624 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657360] rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 02:21:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657360/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657360/ SELinux denials found on ubuntu@smithi002.front.sepia.ceph.com: ['type=AVC msg=audit(1643867084.119:2858): avc: denied { setgid } for pid=15129 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867142.302:2985): avc: denied { setgid } for pid=17647 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867141.540:2981): avc: denied { setgid } for pid=17641 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867083.787:2851): avc: denied { setgid } for pid=15124 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867142.625:2987): avc: denied { setgid } for pid=17655 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867083.121:2847): avc: denied { setgid } for pid=15118 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657364] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_mix} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657364/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657364/ SELinux denials found on ubuntu@smithi154.front.sepia.ceph.com: ['type=AVC msg=audit(1643867186.759:2859): avc: denied { setgid } for pid=15169 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867186.467:2857): avc: denied { setgid } for pid=15155 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867213.383:2865): avc: denied { setgid } for pid=15945 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867185.604:2853): avc: denied { setgid } for pid=15149 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867214.361:2870): avc: denied { setgid } for pid=15956 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657367] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657367/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=20ef9003ffb74a0a9f167448f650632b Command failed on smithi052 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph- ci/ceph:fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c0f056bc-84b4-11ec-8c35-001a4aab830c -- bash -c \'set -e\nset -x\nceph orch ps\nHOST=$(hostname -s)\nOSD=$(ceph orch ps $HOST | grep osd | head -n 1 | awk \'"\'"\'{print $1}\'"\'"\')\necho "host $HOST, osd $OSD"\nceph orch daemon stop $OSD\nwhile ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\nceph auth export $OSD > k\nceph orch daemon rm $OSD --force\nceph orch ps --refresh\nwhile ceph orch ps | grep $OSD ; do sleep 5 ; done\nceph auth add $OSD -i k\nceph cephadm osd activate $HOST\nwhile ! ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\n\'' [6657371] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} ----------------------------------------------------------------- time: 00:15:39 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657371/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657371/ SELinux denials found on ubuntu@smithi040.front.sepia.ceph.com: ['type=AVC msg=audit(1643867339.108:2937): avc: denied { setgid } for pid=16490 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867340.060:2942): avc: denied { setgid } for pid=16500 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657372] rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657372/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68ee4cc2e88443f1bee49e14ba36613b Command failed on smithi146 with status 1: 'sudo yum install -y kernel' [6657374] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep} ----------------------------------------------------------------- time: 00:22:08 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657374/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657374/ SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1643867437.217:2980): avc: denied { setgid } for pid=17623 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867437.538:2982): avc: denied { setgid } for pid=17628 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867374.687:2841): avc: denied { setgid } for pid=15098 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867375.637:2846): avc: denied { setgid } for pid=15108 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867436.593:2976): avc: denied { setgid } for pid=17617 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657375] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} ----------------------------------------------------------------- time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657375/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657375/ SELinux denials found on ubuntu@smithi016.front.sepia.ceph.com: ['type=AVC msg=audit(1643867409.359:2857): avc: denied { setgid } for pid=15132 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867408.306:2852): avc: denied { setgid } for pid=15122 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867475.134:2987): avc: denied { setgid } for pid=17654 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867474.123:2982): avc: denied { setgid } for pid=17643 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657376] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/sync workloads/rados_5925} ----------------------------------------------------------------- time: 00:09:03 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657376/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657376/ SELinux denials found on ubuntu@smithi065.front.sepia.ceph.com: ['type=AVC msg=audit(1643867482.856:2980): avc: denied { setgid } for pid=17626 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867419.212:2844): avc: denied { setgid } for pid=15098 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867420.446:2853): avc: denied { setgid } for pid=15121 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867420.242:2849): avc: denied { setgid } for pid=15108 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867481.854:2978): avc: denied { setgid } for pid=17617 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657377] rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:06:51 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657377/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657377/ SELinux denials found on ubuntu@smithi031.front.sepia.ceph.com: ['type=AVC msg=audit(1643867390.950:2888): avc: denied { setgid } for pid=15269 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867449.724:3023): avc: denied { setgid } for pid=17787 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867450.734:3025): avc: denied { setgid } for pid=17796 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867391.992:2893): avc: denied { setgid } for pid=15279 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657378] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/cache} ----------------------------------------------------------------- time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657378/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657378/ SELinux denials found on ubuntu@smithi038.front.sepia.ceph.com: ['type=AVC msg=audit(1643867492.106:2856): avc: denied { setgid } for pid=15172 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867493.196:2862): avc: denied { setgid } for pid=15182 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867492.893:2860): avc: denied { setgid } for pid=15178 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867558.067:2987): avc: denied { setgid } for pid=17701 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867559.070:2992): avc: denied { setgid } for pid=17712 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657380] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{centos_8} tasks/progress} ----------------------------------------------------------------- time: 00:21:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657380/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657380/ SELinux denials found on ubuntu@smithi071.front.sepia.ceph.com: ['type=AVC msg=audit(1643867421.027:2846): avc: denied { setgid } for pid=15115 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867420.074:2844): avc: denied { setgid } for pid=15107 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867479.544:2974): avc: denied { setgid } for pid=17623 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867480.537:2979): avc: denied { setgid } for pid=17634 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657381] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} ----------------------------------------------------------------- time: 00:08:57 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657381/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657381/ SELinux denials found on ubuntu@smithi029.front.sepia.ceph.com: ['type=AVC msg=audit(1643867508.233:2863): avc: denied { setgid } for pid=15161 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867570.697:2996): avc: denied { setgid } for pid=17688 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867507.424:2859): avc: denied { setgid } for pid=15155 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867569.701:2990): avc: denied { setgid } for pid=17677 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867570.411:2994): avc: denied { setgid } for pid=17683 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867508.523:2865): avc: denied { setgid } for pid=15165 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657384] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:27:06 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657384/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657384/ SELinux denials found on ubuntu@smithi084.front.sepia.ceph.com: ['type=AVC msg=audit(1643867540.026:2984): avc: denied { setgid } for pid=17609 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867539.393:2980): avc: denied { setgid } for pid=17603 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867540.369:2986): avc: denied { setgid } for pid=17614 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867482.715:2852): avc: denied { setgid } for pid=15097 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867481.727:2847): avc: denied { setgid } for pid=15087 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657390] rados/objectstore/{backends/fusestore supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:05:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657390/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657390/ SELinux denials found on ubuntu@smithi070.front.sepia.ceph.com: ['type=AVC msg=audit(1643867770.877:2979): avc: denied { setgid } for pid=17591 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867713.905:2846): avc: denied { setgid } for pid=15067 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867712.874:2841): avc: denied { setgid } for pid=15057 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867769.823:2971): avc: denied { setgid } for pid=17574 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657391] rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:07:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657391/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657391/ SELinux denials found on ubuntu@smithi155.front.sepia.ceph.com: ['type=AVC msg=audit(1643867743.136:2844): avc: denied { setgid } for pid=15067 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867744.362:2853): avc: denied { setgid } for pid=15086 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867744.145:2849): avc: denied { setgid } for pid=15077 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867802.202:2980): avc: denied { setgid } for pid=17595 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867801.268:2978): avc: denied { setgid } for pid=17586 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657392] rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657392/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657392/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b223f667d655447f85e4ec67f2813a8e Command failed on smithi188 with status 1: 'sudo yum install -y kernel' [6657393] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:13 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657393/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657393/ SELinux denials found on ubuntu@smithi124.front.sepia.ceph.com: ['type=AVC msg=audit(1643867823.789:2845): avc: denied { setgid } for pid=15051 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867887.935:2978): avc: denied { setgid } for pid=17576 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867888.862:2983): avc: denied { setgid } for pid=17587 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867824.773:2850): avc: denied { setgid } for pid=15061 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657395] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps} ----------------------------------------------------------------- time: 00:14:38 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657395/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657395/ SELinux denials found on ubuntu@smithi150.front.sepia.ceph.com: ['type=AVC msg=audit(1643867930.304:2980): avc: denied { setgid } for pid=17606 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867873.514:2845): avc: denied { setgid } for pid=15079 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867874.507:2847): avc: denied { setgid } for pid=15087 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643867929.334:2975): avc: denied { setgid } for pid=17595 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657398] rados/cephadm/workunits/{agent/off mon_election/classic task/test_cephadm} ----------------------------------------------------------------- time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657398/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9fe595e93a44bfab956fb88e00bff53 Command failed (workunit test cephadm/test_cephadm.sh) on smithi006 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh' [6657403] rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/dashboard} ----------------------------------------------------------------- time: 01:12:04 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657403/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657403/ SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1643868082.380:2902): avc: denied { setgid } for pid=16368 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868081.305:2900): avc: denied { setgid } for pid=16360 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868146.703:3033): avc: denied { setgid } for pid=18244 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868147.689:3038): avc: denied { setgid } for pid=18255 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657406] rados/upgrade/parallel/{0-random-distro$/{centos_8.3_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657406/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657406/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e524dedbe3eb4c208d06ef0afffdf12b Command failed on smithi039 with status 1: 'sudo yum install -y kernel' [6657413] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/rados_api_tests} ----------------------------------------------------------------- time: 00:45:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657413/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657413/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1de2a7c0d8d2418a90e50e1abe1d8112 Command failed (workunit test rados/test.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6657418] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.2_container_tools_3.0} 1-start 2-services/basic 3-final} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657418/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4f93ed3614b4070a401706ac78d8d08 Command failed on smithi104 with status 1: 'sudo yum install -y kernel' [6657419] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{centos_8} tasks/prometheus} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657419/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657419/ SELinux denials found on ubuntu@smithi067.front.sepia.ceph.com: ['type=AVC msg=audit(1643868495.155:2983): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868422.464:2845): avc: denied { setgid } for pid=15121 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868423.502:2850): avc: denied { setgid } for pid=15131 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868494.186:2978): avc: denied { setgid } for pid=17639 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657420] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:30:52 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657420/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657420/ SELinux denials found on ubuntu@smithi029.front.sepia.ceph.com: ['type=AVC msg=audit(1643868432.132:2887): avc: denied { setgid } for pid=16394 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868433.191:2892): avc: denied { setgid } for pid=16405 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657425] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657425/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657425/ SELinux denials found on ubuntu@smithi057.front.sepia.ceph.com: ['type=AVC msg=audit(1643868414.807:2844): avc: denied { setgid } for pid=15123 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868479.157:2981): avc: denied { setgid } for pid=17645 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868479.428:2983): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868415.996:2846): avc: denied { setgid } for pid=15131 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868478.392:2977): avc: denied { setgid } for pid=17639 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657427] rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:05:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657427/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657427/ SELinux denials found on ubuntu@smithi138.front.sepia.ceph.com: ['type=AVC msg=audit(1643868463.972:2840): avc: denied { setgid } for pid=15068 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868523.598:2979): avc: denied { setgid } for pid=17599 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868522.564:2974): avc: denied { setgid } for pid=17587 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868464.955:2845): avc: denied { setgid } for pid=15078 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868465.162:2849): avc: denied { setgid } for pid=15087 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657428] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:26:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657428/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657428/ SELinux denials found on ubuntu@smithi166.front.sepia.ceph.com: ['type=AVC msg=audit(1643868534.695:2975): avc: denied { setgid } for pid=17644 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868477.734:2845): avc: denied { setgid } for pid=15127 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868478.651:2850): avc: denied { setgid } for pid=15137 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868536.016:2980): avc: denied { setgid } for pid=17665 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657430] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/radosbench} ----------------------------------------------------------------- time: 01:11:13 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657430/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657430/ SELinux denials found on ubuntu@smithi064.front.sepia.ceph.com: ['type=AVC msg=audit(1643868491.491:2846): avc: denied { setgid } for pid=15114 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868552.170:2984): avc: denied { setgid } for pid=17640 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868551.144:2979): avc: denied { setgid } for pid=17629 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868492.787:2851): avc: denied { setgid } for pid=15134 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657431] rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657431/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657431/ sentry: https://sentry.ceph.com/organizations/ceph/?query=62bfc12a01724547a97653bec4d533e3 Command failed on smithi070 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend' [6657434] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_recovery} ----------------------------------------------------------------- time: 00:10:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657434/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657434/ SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1643868621.645:2847): avc: denied { setgid } for pid=15110 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868622.736:2849): avc: denied { setgid } for pid=15118 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868686.293:2977): avc: denied { setgid } for pid=17626 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868687.262:2982): avc: denied { setgid } for pid=17637 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868687.464:2986): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657437] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657437/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657437/ SELinux denials found on ubuntu@smithi172.front.sepia.ceph.com: ['type=AVC msg=audit(1643868766.379:2986): avc: denied { setgid } for pid=17655 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868707.213:2848): avc: denied { setgid } for pid=15128 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868765.401:2981): avc: denied { setgid } for pid=17644 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868708.202:2853): avc: denied { setgid } for pid=15138 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657439] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep} ----------------------------------------------------------------- time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657439/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657439/ SELinux denials found on ubuntu@smithi041.front.sepia.ceph.com: ['type=AVC msg=audit(1643868807.245:2991): avc: denied { setgid } for pid=17672 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868806.200:2985): avc: denied { setgid } for pid=17661 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868748.261:2857): avc: denied { setgid } for pid=15152 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868747.205:2852): avc: denied { setgid } for pid=15142 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868806.964:2989): avc: denied { setgid } for pid=17667 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657443] rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:05:48 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657443/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657443/ SELinux denials found on ubuntu@smithi104.front.sepia.ceph.com: ['type=AVC msg=audit(1643868751.604:2846): avc: denied { setgid } for pid=15134 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868812.083:2976): avc: denied { setgid } for pid=17650 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868813.002:2981): avc: denied { setgid } for pid=17661 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868752.625:2848): avc: denied { setgid } for pid=15142 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657444] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} ----------------------------------------------------------------- time: 00:20:58 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657444/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657444/ SELinux denials found on ubuntu@smithi074.front.sepia.ceph.com: ['type=AVC msg=audit(1643868832.912:2978): avc: denied { setgid } for pid=17632 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868773.986:2848): avc: denied { setgid } for pid=15116 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868833.865:2983): avc: denied { setgid } for pid=17643 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868774.927:2850): avc: denied { setgid } for pid=15124 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657446] rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-wait} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657446/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657446/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68ae6938057348cdafa0b4591e2bf067 Command failed on smithi038 with status 1: 'sudo yum install -y kernel' [6657449] rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:21:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657449/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657449/ SELinux denials found on ubuntu@smithi135.front.sepia.ceph.com: ['type=AVC msg=audit(1643869030.812:2997): avc: denied { setgid } for pid=17627 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868971.751:2862): avc: denied { setgid } for pid=15100 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869029.376:2992): avc: denied { setgid } for pid=17616 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643868972.704:2867): avc: denied { setgid } for pid=15110 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657454] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object} ----------------------------------------------------------------- time: 00:10:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657454/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657454/ SELinux denials found on ubuntu@smithi071.front.sepia.ceph.com: ['type=AVC msg=audit(1643869248.969:2865): avc: denied { setgid } for pid=15123 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869314.941:2995): avc: denied { setgid } for pid=17645 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869313.992:2993): avc: denied { setgid } for pid=17636 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869248.566:2863): avc: denied { setgid } for pid=15119 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869247.896:2859): avc: denied { setgid } for pid=15113 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657456] rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657456/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657456/ SELinux denials found on ubuntu@smithi146.front.sepia.ceph.com: ['type=AVC msg=audit(1643869339.921:2986): avc: denied { setgid } for pid=17619 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869281.025:2847): avc: denied { setgid } for pid=15092 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869339.654:2984): avc: denied { setgid } for pid=17606 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869338.985:2980): avc: denied { setgid } for pid=17600 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869282.005:2852): avc: denied { setgid } for pid=15102 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657460] rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} ----------------------------------------------------------------- time: 00:17:15 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657460/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657460/ SELinux denials found on ubuntu@smithi036.front.sepia.ceph.com: ['type=AVC msg=audit(1643869334.190:2846): avc: denied { setgid } for pid=15121 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869392.422:2984): avc: denied { setgid } for pid=17648 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869335.192:2851): avc: denied { setgid } for pid=15131 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869391.378:2979): avc: denied { setgid } for pid=17637 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657461] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:21:15 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657461/ SELinux denials found on ubuntu@smithi037.front.sepia.ceph.com: ['type=AVC msg=audit(1643869451.512:2981): avc: denied { setgid } for pid=17645 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869395.909:2853): avc: denied { setgid } for pid=15136 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869394.906:2851): avc: denied { setgid } for pid=15128 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869452.531:2986): avc: denied { setgid } for pid=17656 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657464] rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} ----------------------------------------------------------------- time: 00:12:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657464/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657464/ SELinux denials found on ubuntu@smithi039.front.sepia.ceph.com: ['type=AVC msg=audit(1643869654.342:3037): avc: denied { setgid } for pid=18259 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869653.323:3032): avc: denied { setgid } for pid=18248 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869597.828:2903): avc: denied { setgid } for pid=16365 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869596.793:2898): avc: denied { setgid } for pid=16355 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657465] rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657465/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc23be30193045b6a6e2e4e4d35349a4 Command failed on smithi104 with status 1: 'sudo yum install -y kernel' [6657467] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews} ----------------------------------------------------------------- time: 00:06:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657467/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657467/ SELinux denials found on ubuntu@smithi046.front.sepia.ceph.com: ['type=AVC msg=audit(1643869718.092:2977): avc: denied { setgid } for pid=17611 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869658.166:2841): avc: denied { setgid } for pid=15080 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869717.088:2971): avc: denied { setgid } for pid=17600 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869717.800:2975): avc: denied { setgid } for pid=17606 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869659.166:2846): avc: denied { setgid } for pid=15090 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657472] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:20:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657472/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657472/ SELinux denials found on ubuntu@smithi083.front.sepia.ceph.com: ['type=AVC msg=audit(1643869772.592:2996): avc: denied { setgid } for pid=17618 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869714.728:2862): avc: denied { setgid } for pid=15098 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869715.480:2866): avc: denied { setgid } for pid=15104 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869773.851:3000): avc: denied { setgid } for pid=17624 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869774.204:3002): avc: denied { setgid } for pid=17639 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869715.756:2868): avc: denied { setgid } for pid=15108 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657473] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} ----------------------------------------------------------------- time: 06:23:41 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657473/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6bc38d85df8498aa585fc054b54ce16 Command failed (workunit test rados/test.sh) on smithi041 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fc4f59a13b5cfe31b1d75dc7e873c108d04bff91 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6657474] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/one workloads/rados_mon_workunits} ----------------------------------------------------------------- time: 00:26:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657474/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657474/ SELinux denials found on ubuntu@smithi164.front.sepia.ceph.com: ['type=AVC msg=audit(1643869875.886:2982): avc: denied { setgid } for pid=17639 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869819.136:2847): avc: denied { setgid } for pid=15112 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869874.900:2977): avc: denied { setgid } for pid=17628 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869820.079:2852): avc: denied { setgid } for pid=15122 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657480] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} ----------------------------------------------------------------- time: 00:31:58 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657480/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657480/ SELinux denials found on ubuntu@smithi170.front.sepia.ceph.com: ['type=AVC msg=audit(1643869969.505:2848): avc: denied { setgid } for pid=15133 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870031.712:2983): avc: denied { setgid } for pid=17660 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870030.700:2978): avc: denied { setgid } for pid=17649 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643869970.495:2853): avc: denied { setgid } for pid=15143 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657485] rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} ----------------------------------------------------------------- time: 00:17:30 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657485/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657485/ SELinux denials found on ubuntu@smithi101.front.sepia.ceph.com: ['type=AVC msg=audit(1643870166.700:2908): avc: denied { setgid } for pid=16358 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870222.901:3041): avc: denied { setgid } for pid=18247 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870223.836:3046): avc: denied { setgid } for pid=18258 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870167.698:2913): avc: denied { setgid } for pid=16368 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657486] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{centos_8} tasks/crash} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657486/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657486/ SELinux denials found on ubuntu@smithi106.front.sepia.ceph.com: ['type=AVC msg=audit(1643870236.517:2976): avc: denied { setgid } for pid=17630 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870180.519:2846): avc: denied { setgid } for pid=15114 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870181.458:2848): avc: denied { setgid } for pid=15122 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870238.021:2981): avc: denied { setgid } for pid=17653 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657491] rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657491/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74ecd255ab8f4ac7825738cf552022e4 Command failed on smithi159 with status 1: 'sudo yum install -y kernel' [6657494] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} ----------------------------------------------------------------- time: 00:25:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657494/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657494/ SELinux denials found on ubuntu@smithi092.front.sepia.ceph.com: ['type=AVC msg=audit(1643870429.713:2854): avc: denied { setgid } for pid=15093 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870428.595:2848): avc: denied { setgid } for pid=15080 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870518.235:2984): avc: denied { setgid } for pid=17613 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870429.294:2852): avc: denied { setgid } for pid=15086 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870517.241:2979): avc: denied { setgid } for pid=17602 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657498] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} ----------------------------------------------------------------- time: 00:24:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657498/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657498/ SELinux denials found on ubuntu@smithi204.front.sepia.ceph.com: ['type=AVC msg=audit(1643870577.475:2980): avc: denied { setgid } for pid=17584 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870518.286:2847): avc: denied { setgid } for pid=15066 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870578.495:2986): avc: denied { setgid } for pid=17596 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870578.183:2984): avc: denied { setgid } for pid=17590 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870519.315:2852): avc: denied { setgid } for pid=15076 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657504] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados tasks/rados_cls_all validater/lockdep} ----------------------------------------------------------------- time: 00:15:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657504/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657504/ SELinux denials found on ubuntu@smithi055.front.sepia.ceph.com: ['type=AVC msg=audit(1643870656.143:2993): avc: denied { setgid } for pid=17677 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870596.082:2855): avc: denied { setgid } for pid=15138 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870654.674:2988): avc: denied { setgid } for pid=17656 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870597.136:2860): avc: denied { setgid } for pid=15148 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] [6657507] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/snaps-few-objects} ----------------------------------------------------------------- time: 00:27:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657507/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657507/ SELinux denials found on ubuntu@smithi134.front.sepia.ceph.com: ['type=AVC msg=audit(1643870673.227:2857): avc: denied { setgid } for pid=15111 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870672.295:2855): avc: denied { setgid } for pid=15103 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870729.495:2990): avc: denied { setgid } for pid=17646 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tclass=capability permissive=1', 'type=AVC msg=audit(1643870728.226:2985): avc: denied { setgid } for pid=17625 comm="sss_cache" capability=6 scontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:groupadd_t:s0-s0:c0.c1023 tclass=capability permissive=1'] Pass ================================================================= [6657183] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:10:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657183/ [6657184] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:08:49 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657184/ [6657188] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:09:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657188/ [6657190] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:15:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657190/ [6657192] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 01:01:25 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657192/ [6657195] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:20:03 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657195/ [6657196] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:32:54 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657196/ [6657198] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657198/ [6657202] rados/standalone/{supported-random-distro$/{rhel_8} workloads/scrub} time: 02:28:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657202/ [6657204] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:07:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657204/ [6657205] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657205/ [6657211] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/rados_api_tests} time: 00:30:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657211/ [6657214] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start} time: 00:19:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657214/ [6657215] rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657215/ [6657216] rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}} time: 00:22:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657216/ [6657218] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench} time: 01:10:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657218/ [6657220] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:09:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657220/ [6657221] rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:41:42 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657221/ [6657222] rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} time: 00:12:25 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657222/ [6657223] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:34:08 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657223/ [6657224] rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_orch_cli} time: 00:13:10 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657224/ [6657225] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect} time: 00:10:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657225/ [6657226] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} time: 00:09:22 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657226/ [6657228] rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{rhel_8}} time: 00:25:54 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657228/ [6657229] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:34:49 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657229/ [6657230] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} time: 00:16:30 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657230/ [6657232] rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:28:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657232/ [6657233] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests} time: 00:10:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657233/ [6657234] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} time: 00:20:49 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657234/ [6657235] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:09:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657235/ [6657236] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} tasks/mon_recovery} time: 00:35:54 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657236/ [6657239] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/crash} time: 00:08:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657239/ [6657240] rados/objectstore/{backends/alloc-hint supported-random-distro$/{rhel_8}} time: 00:22:02 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657240/ [6657241] rados/rest/{mgr-restful supported-random-distro$/{rhel_8}} time: 00:24:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657241/ [6657242] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/calico rook/1.7.2} time: 00:31:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657242/ [6657245] rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} time: 01:29:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657245/ [6657247] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657247/ [6657249] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_workunits} time: 00:25:07 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657249/ [6657253] rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:07:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657253/ [6657254] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/basic 3-final} time: 00:17:32 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657254/ [6657255] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:08:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657255/ [6657256] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} time: 00:11:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657256/ [6657257] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} time: 00:12:34 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657257/ [6657258] rados/cephadm/workunits/{agent/off mon_election/classic task/test_adoption} time: 00:07:03 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657258/ [6657260] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} time: 00:37:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657260/ [6657262] rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:08:47 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657262/ [6657267] rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}} time: 00:26:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657267/ [6657268] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657268/ [6657270] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized} time: 00:32:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657270/ [6657271] rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{rhel_8}} time: 00:24:49 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657271/ [6657272] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/mon_clock_no_skews} time: 00:21:32 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657272/ [6657273] rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:35:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657273/ [6657274] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} time: 00:09:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657274/ [6657276] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{rhel_8} tasks/failover} time: 00:25:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657276/ [6657277] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/many workloads/rados_mon_workunits} time: 00:41:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657277/ [6657278] rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:19:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657278/ [6657279] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657279/ [6657280] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects} time: 00:18:31 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657280/ [6657282] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} time: 00:58:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657282/ [6657283] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657283/ [6657285] rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:06:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657285/ [6657286] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:40:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657286/ [6657287] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:38:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657287/ [6657288] rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:44 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657288/ [6657289] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:10:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657289/ [6657291] rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:07:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657291/ [6657292] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} time: 00:26:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657292/ [6657293] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/repave-all} time: 00:12:38 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657293/ [6657294] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} time: 00:38:58 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657294/ [6657295] rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:05:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657295/ [6657296] rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657296/ [6657299] rados/cephadm/workunits/{agent/off mon_election/classic task/test_cephadm_repos} time: 00:04:10 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657299/ [6657300] rados/objectstore/{backends/filejournal supported-random-distro$/{ubuntu_latest}} time: 00:06:13 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657300/ [6657302] rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:20:08 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657302/ [6657303] rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:22:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657303/ [6657304] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} time: 00:34:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657304/ [6657305] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:10:28 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657305/ [6657306] rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:14:15 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657306/ [6657308] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/one workloads/snaps-few-objects} time: 00:48:46 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657308/ [6657309] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed} time: 00:26:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657309/ [6657310] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{filestore-xfs} supported-random-distro$/{ubuntu_latest} tasks/insights} time: 00:11:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657310/ [6657311] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657311/ [6657314] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:43:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657314/ [6657316] rados/standalone/{supported-random-distro$/{rhel_8} workloads/erasure-code} time: 01:01:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657316/ [6657317] rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}} time: 01:11:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657317/ [6657318] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:22:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657318/ [6657319] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:19:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657319/ [6657321] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} time: 00:09:55 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657321/ [6657323] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/flannel rook/master} time: 00:16:20 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657323/ [6657324] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:06:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657324/ [6657325] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-radosbench} time: 01:16:48 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657325/ [6657327] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs} time: 00:27:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657327/ [6657329] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big} time: 00:32:21 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657329/ [6657330] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:22:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657330/ [6657331] rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools} 1-start 2-services/rgw 3-final} time: 00:09:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657331/ [6657336] rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} time: 00:12:23 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657336/ [6657337] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-agent-small} time: 00:24:44 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657337/ [6657338] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_big} time: 00:33:30 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657338/ [6657341] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:20:30 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657341/ [6657343] rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} time: 00:08:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657343/ [6657344] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/sync-many workloads/pool-create-delete} time: 00:27:45 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657344/ [6657345] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:54 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657345/ [6657347] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/module_selftest} time: 00:25:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657347/ [6657348] rados/cephadm/workunits/{agent/off mon_election/classic task/test_orch_cli} time: 00:13:29 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657348/ [6657349] rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} time: 00:06:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657349/ [6657352] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mgr} time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657352/ [6657354] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} time: 00:45:04 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657354/ [6657355] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} time: 00:23:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657355/ [6657356] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} time: 00:09:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657356/ [6657357] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} time: 00:17:45 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657357/ [6657358] rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:06:04 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657358/ [6657359] rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:21:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657359/ [6657361] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:17:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657361/ [6657362] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced} time: 00:16:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657362/ [6657363] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:31:55 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657363/ [6657365] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} time: 00:20:20 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657365/ [6657366] rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:28:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657366/ [6657368] rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:15:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657368/ [6657369] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} time: 00:06:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657369/ [6657370] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps} time: 00:21:29 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657370/ [6657373] rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:21:52 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657373/ [6657379] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_adoption} time: 00:07:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657379/ [6657382] rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}} time: 00:07:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657382/ [6657383] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all} time: 00:18:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657383/ [6657385] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} time: 00:33:23 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657385/ [6657386] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed} time: 00:24:03 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657386/ [6657387] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread} time: 00:29:54 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657387/ [6657388] rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657388/ [6657389] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read} time: 00:19:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657389/ [6657394] rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} time: 00:24:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657394/ [6657396] rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657396/ [6657397] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} time: 00:22:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657397/ [6657399] rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:22:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657399/ [6657400] rados/multimon/{clusters/3 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} time: 00:05:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657400/ [6657401] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:10:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657401/ [6657402] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects} time: 00:41:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657402/ [6657404] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/host rook/1.7.2} time: 00:31:40 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657404/ [6657405] rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:29:47 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657405/ [6657407] rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}} time: 00:56:33 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657407/ [6657408] rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} time: 02:28:38 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657408/ [6657409] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657409/ [6657410] rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/rados_api_tests} time: 00:39:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657410/ [6657411] rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:18:20 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657411/ [6657412] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} time: 00:09:31 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657412/ [6657414] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/readwrite} time: 00:10:40 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657414/ [6657415] rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:07:40 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657415/ [6657416] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} time: 00:35:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657416/ [6657417] rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 01:05:56 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657417/ [6657421] rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} time: 00:13:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657421/ [6657422] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency} time: 00:28:38 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657422/ [6657423] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-many-deletes} time: 00:38:32 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657423/ [6657424] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_cephadm_repos} time: 00:04:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657424/ [6657426] rados/standalone/{supported-random-distro$/{rhel_8} workloads/mon} time: 00:59:34 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657426/ [6657429] rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:10:26 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657429/ [6657432] rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657432/ [6657433] rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} time: 00:11:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657433/ [6657435] rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657435/ [6657436] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} time: 00:14:23 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657436/ [6657438] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:47 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657438/ [6657440] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:19:39 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657440/ [6657441] rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} time: 00:56:55 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657441/ [6657442] rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} time: 00:50:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657442/ [6657445] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} time: 00:13:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657445/ [6657447] rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} time: 00:23:57 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657447/ [6657448] rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}} time: 00:20:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657448/ [6657450] rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:48 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657450/ [6657451] rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/workunits} time: 00:07:00 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657451/ [6657452] rados/cephadm/workunits/{agent/off mon_election/classic task/test_nfs} time: 00:22:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657452/ [6657453] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects} time: 00:32:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657453/ [6657455] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} time: 00:11:59 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657455/ [6657457] rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/classic start} time: 00:20:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657457/ [6657458] rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:20:27 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657458/ [6657459] rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd-backfill} time: 02:05:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657459/ [6657462] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} time: 00:09:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657462/ [6657463] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/set-chunks-read} time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657463/ [6657466] rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}} time: 00:24:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657466/ [6657468] rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:45 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657468/ [6657469] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:24:29 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657469/ [6657470] rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:28:41 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657470/ [6657471] rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_orch_cli} time: 00:13:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657471/ [6657475] rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}} time: 00:22:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657475/ [6657476] rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} time: 00:17:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657476/ [6657477] rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:21:40 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657477/ [6657478] rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} time: 00:08:36 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657478/ [6657479] rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{rhel_8}} time: 02:30:11 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657479/ [6657481] rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-localized} time: 00:18:43 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657481/ [6657482] rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} time: 00:08:05 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657482/ [6657483] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/repave-all} time: 00:09:04 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657483/ [6657484] rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} time: 00:43:14 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657484/ [6657487] rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/master} time: 00:18:17 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657487/ [6657488] rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}} time: 00:06:34 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657488/ [6657489] rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657489/ [6657490] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects} time: 00:20:53 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657490/ [6657492] rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} time: 00:36:24 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657492/ [6657493] rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}} time: 00:12:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657493/ [6657495] rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 1-start 2-services/rgw 3-final} time: 00:18:18 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657495/ [6657496] rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{rhel_8}} time: 00:22:07 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657496/ [6657497] rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} time: 01:55:37 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657497/ [6657499] rados/singleton/{all/mon-config mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8}} time: 00:21:16 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657499/ [6657500] rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/mon_clock_with_skews} time: 00:21:06 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657500/ [6657501] rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} time: 00:09:13 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657501/ [6657502] rados/cephadm/workunits/{agent/on mon_election/classic task/test_adoption} time: 00:07:01 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657502/ [6657503] rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} time: 00:23:31 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657503/ [6657505] rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{rhel_8}} time: 00:25:50 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657505/ [6657506] rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized} time: 00:25:09 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657506/ [6657508] rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}} time: 00:46:19 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657508/ [6657509] rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-add} time: 00:09:12 info: http://pulpito.ceph.com/yuriw-2022-02-03_02:33:02-rados-wip-yuri8-testing-2022-02-02-1557-quincy-distro-default-smithi/6657509/