Found 8 jobs Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f0c1f6ed652] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0c1f6ed82d] Ignoring malparsed backtrace: 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.856351", Ignoring malparsed backtrace: 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:28.410791", , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:11.431850", , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:52.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:28.593287", Ignoring malparsed backtrace: 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:37.693947", , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.717939", , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.729941", Ignoring malparsed backtrace: 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.854204", Ignoring malparsed backtrace: 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.969023", , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.772 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.972324", , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:52.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:23.390225", Ignoring malparsed backtrace: 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.801 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0098285809999999994, , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2020-10-27T04:08:52.853 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.220027107999996, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087482700000000007, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { Ignoring malparsed backtrace: 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2020-10-27 04:08:34.643724", , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.088702560000002, , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074977419999999999, Ignoring malparsed backtrace: 2020-10-27T04:08:52.899 INFO:teuthology.orchestra.run.smithi092.stdout:} , 2020-10-27T04:08:52.900 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight , nodeep-scrub is set Ignoring malparsed backtrace: 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:53.203 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.316031", Ignoring malparsed backtrace: 2020-10-27T04:08:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.880210", , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read" , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.851339", Ignoring malparsed backtrace: 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.948591", , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.545996", , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered" , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.961007", Ignoring malparsed backtrace: 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.570838", , 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched" , 2020-10-27T04:08:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.910208", Ignoring malparsed backtrace: 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.576005", , 2020-10-27T04:08:53.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.882882", Ignoring malparsed backtrace: 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.903080", , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" Ignoring malparsed backtrace: 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 645 , 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.822 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.026680145999999998, , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2020-10-27T04:08:53.862 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.891062", , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:50.383624", Ignoring malparsed backtrace: 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5225 , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:09.403819", , 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:53.931 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", Ignoring malparsed backtrace: 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.479655", Ignoring malparsed backtrace: 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.480291", , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:54.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.854040", Ignoring malparsed backtrace: 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.399499", , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.938560", Ignoring malparsed backtrace: 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.404750", , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.409309", , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered" , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4772 Ignoring malparsed backtrace: 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:29.380947", , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:37.399577", , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched" , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:11.433446", Ignoring malparsed backtrace: 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.511196", , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ Ignoring malparsed backtrace: 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.866526", , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.888348", Ignoring malparsed backtrace: 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.918309", , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.992 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917155", , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4570 Ignoring malparsed backtrace: 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:18.059358", , 2020-10-27T04:08:55.035 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:32.434336", , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ , 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.076 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:23.573555", Ignoring malparsed backtrace: 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:34.744493", , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.823720", , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917821", Ignoring malparsed backtrace: 2020-10-27T04:08:55.516 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.883656", , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:46.509124", , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done" , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.559 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.675162", , 2020-10-27T04:08:55.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.720798", , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.615 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.691906", , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5449 , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:09:13.013 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":142,"fsid":"4c43081c-176b-4fed-8aad-53b3ee89c837","created":"2020-10-27 04:06:32.593739","modified":"2020-10-27 04:09:12.969268","last_up_change":"2020-10-27 04:06:41.641767","last_in_change":"2020-10-27 04:07:25.367025","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-10-27 04:06:45.894327","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"90","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2020-10-27 04:07:02.815119","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":99,"snap_epoch":124,"pool_snaps":[],"removed_snaps":"[1~2d,2f~a,3a~2,3d~4,45~4,4a~7,53~5,59~6]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1df688f3-1d4d-4143-afcb-b4db528993bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":14262},{"type":"v1","addr":"172.21.15.92:6825","nonce":14262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":14262},{"type":"v1","addr":"172.21.15.92:6827","nonce":14262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":14262},{"type":"v1","addr":"172.21.15.92:6831","nonce":14262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":14262},{"type":"v1","addr":"172.21.15.92:6829","nonce":14262}]},"public_addr":"172.21.15.92:6825/14262","cluster_addr":"172.21.15.92:6827/14262","heartbeat_back_addr":"172.21.15.92:6831/14262","heartbeat_front_addr":"172.21.15.92:6829/14262","state":["exists","up"]},{"osd":1,"uuid":"d3f3dd2b-1944-4e9a-b751-f370a9f34f56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":14263},{"type":"v1","addr":"172.21.15.92:6810","nonce":14263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":14263},{"type":"v1","addr":"172.21.15.92:6814","nonce":14263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":14263},{"type":"v1","addr":"172.21.15.92:6823","nonce":14263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":14263},{"type":"v1","addr":"172.21.15.92:6819","nonce":14263}]},"public_addr":"172.21.15.92:6810/14263","cluster_addr":"172.21.15.92:6814/14263","heartbeat_back_addr":"172.21.15.92:6823/14263","heartbeat_front_addr":"172.21.15.92:6819/14263","state":["exists","up"]},{"osd":2,"uuid":"f13ee95a-f9be-4695-a412-e444a664d608","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":14264},{"type":"v1","addr":"172.21.15.92:6811","nonce":14264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":14264},{"type":"v1","addr":"172.21.15.92:6815","nonce":14264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":14264},{"type":"v1","addr":"172.21.15.92:6822","nonce":14264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":14264},{"type":"v1","addr":"172.21.15.92:6818","nonce":14264}]},"public_addr":"172.21.15.92:6811/14264","cluster_addr":"172.21.15.92:6815/14264","heartbeat_back_addr":"172.21.15.92:6822/14264","heartbeat_front_addr":"172.21.15.92:6818/14264","state":["exists","up"]},{"osd":3,"uuid":"6fbdb1f1-829a-4bf1-a5c4-2973d47a9fb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":14261},{"type":"v1","addr":"172.21.15.92:6801","nonce":14261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":14261},{"type":"v1","addr":"172.21.15.92:6803","nonce":14261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":14261},{"type":"v1","addr":"172.21.15.92:6807","nonce":14261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":14261},{"type":"v1","addr":"172.21.15.92:6805","nonce":14261}]},"public_addr":"172.21.15.92:6801/14261","cluster_addr":"172.21.15.92:6803/14261","heartbeat_back_addr":"172.21.15.92:6807/14261","heartbeat_front_addr":"172.21.15.92:6805/14261","state":["exists","up"]},{"osd":4,"uuid":"b88aa06a-d50b-497a-b8bb-578f3bd7585a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":14407},{"type":"v1","addr":"172.21.15.143:6819","nonce":14407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":14407},{"type":"v1","addr":"172.21.15.143:6821","nonce":14407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":14407},{"type":"v1","addr":"172.21.15.143:6825","nonce":14407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":14407},{"type":"v1","addr":"172.21.15.143:6823","nonce":14407}]},"public_addr":"172.21.15.143:6819/14407","cluster_addr":"172.21.15.143:6821/14407","heartbeat_back_addr":"172.21.15.143:6825/14407","heartbeat_front_addr":"172.21.15.143:6823/14407","state":["exists","up"]},{"osd":5,"uuid":"70ee976a-1c73-4a16-9772-ccb0b29ce83d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":14431},{"type":"v1","addr":"172.21.15.143:6827","nonce":14431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":14431},{"type":"v1","addr":"172.21.15.143:6829","nonce":14431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":14431},{"type":"v1","addr":"172.21.15.143:6833","nonce":14431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":14431},{"type":"v1","addr":"172.21.15.143:6831","nonce":14431}]},"public_addr":"172.21.15.143:6827/14431","cluster_addr":"172.21.15.143:6829/14431","heartbeat_back_addr":"172.21.15.143:6833/14431","heartbeat_front_addr":"172.21.15.143:6831/14431","state":["exists","up"]},{"osd":6,"uuid":"b5af2580-ae4b-4740-b443-d5e345375840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":14416},{"type":"v1","addr":"172.21.15.143:6811","nonce":14416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":14416},{"type":"v1","addr":"172.21.15.143:6813","nonce":14416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":14416},{"type":"v1","addr":"172.21.15.143:6817","nonce":14416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":14416},{"type":"v1","addr":"172.21.15.143:6815","nonce":14416}]},"public_addr":"172.21.15.143:6811/14416","cluster_addr":"172.21.15.143:6813/14416","heartbeat_back_addr":"172.21.15.143:6817/14416","heartbeat_front_addr":"172.21.15.143:6815/14416","state":["exists","up"]},{"osd":7,"uuid":"6c19afa9-e887-422e-8af6-bd96527de403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":14408},{"type":"v1","addr":"172.21.15.143:6803","nonce":14408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":14408},{"type":"v1","addr":"172.21.15.143:6805","nonce":14408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":14408},{"type":"v1","addr":"172.21.15.143:6809","nonce":14408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":14408},{"type":"v1","addr":"172.21.15.143:6807","nonce":14408}]},"public_addr":"172.21.15.143:6803/14408","cluster_addr":"172.21.15.143:6805/14408","heartbeat_back_addr":"172.21.15.143:6809/14408","heartbeat_front_addr":"172.21.15.143:6807/14408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":24,"length":1},{"begin":36,"length":1},{"begin":39,"length":1},{"begin":42,"length":1},{"begin":48,"length":3},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":69,"length":1},{"begin":72,"length":1},{"begin":74,"length":1},{"begin":77,"length":3},{"begin":85,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} Ignoring malparsed backtrace: 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ , 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v2", Ignoring malparsed backtrace: 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", Ignoring malparsed backtrace: 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "upmap_max_iterations", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.009 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_USER_ID": { , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_USER_ID", Ignoring malparsed backtrace: 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "deepsea", , 2020-10-27T04:09:20.067 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "device_health_metrics", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.095 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "diskprediction_default_authority", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "43200", , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "name prefix for any created local pool", , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, , 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "error_string": "", , 2020-10-27T04:09:20.169 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.198 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.226 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: "channel_crash": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "interval", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "identifier": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "identifier", Ignoring malparsed backtrace: 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "smtp_destination", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "beginning time of day to automatically balance", Ignoring malparsed backtrace: 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "secs", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "FEATURE_TOGGLE_mirroring": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "GRAFANA_API_USERNAME", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.385 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Enable/disable debug options", Ignoring malparsed backtrace: 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.414 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "self_heal": { , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "self_heal", Ignoring malparsed backtrace: 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.444 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "password", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "failure_domain": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "failure_domain", Ignoring malparsed backtrace: 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "rwoption5": { , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "rwoption5", Ignoring malparsed backtrace: 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: Set osd_debug_deep_scrub_sleep to 0 , 2020-10-27T04:09:25.675 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:10:07.915 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:07:57.387753 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.189 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:50.069 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.11 , instructing pg 1.11 on osd.6 to deep-scrub , 2020-10-27T04:11:50.484 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 Ignoring malparsed backtrace: 2020-10-27T04:12:22.404 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:13:23.752 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 2.cs0 on osd.6 to deep-scrub , 2020-10-27T04:14:07.859 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.8 , instructing pg 1.8 on osd.1 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:14:56.987 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.345 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:16:25.121 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.6 , instructing pg 2.6s0 on osd.4 to deep-scrub , 2020-10-27T04:16:25.501 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.4 Ignoring malparsed backtrace: 2020-10-27T04:17:11.077 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:17:11.078 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:17:31.079 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:18:12.385 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 1.1 on osd.3 to deep-scrub , 2020-10-27T04:18:42.606 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.2 , instructing pg 2.2s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:19:45.623 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.624 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.625 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.915 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:18.628 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:22:19.518 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , 2020-10-27T04:22:19.899 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all Ignoring malparsed backtrace: 2020-10-27T04:23:00.748 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.378 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:07:56.847884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.362 DEBUG:tasks.ceph.osd.4:waiting for process to exit , 2020-10-27T04:24:54.362 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:24:54.428 INFO:tasks.ceph.osd.4:Stopped Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f0c1f6ed652] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0c1f6ed82d] Ignoring malparsed backtrace: 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.856351", Ignoring malparsed backtrace: 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:28.410791", , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:11.431850", , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:52.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:28.593287", Ignoring malparsed backtrace: 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:37.693947", , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.717939", , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.729941", Ignoring malparsed backtrace: 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.854204", Ignoring malparsed backtrace: 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.969023", , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.772 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.972324", , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:52.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:23.390225", Ignoring malparsed backtrace: 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.801 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0098285809999999994, , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2020-10-27T04:08:52.853 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.220027107999996, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087482700000000007, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { Ignoring malparsed backtrace: 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2020-10-27 04:08:34.643724", , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.088702560000002, , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074977419999999999, Ignoring malparsed backtrace: 2020-10-27T04:08:52.899 INFO:teuthology.orchestra.run.smithi092.stdout:} , 2020-10-27T04:08:52.900 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight , nodeep-scrub is set Ignoring malparsed backtrace: 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:53.203 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.316031", Ignoring malparsed backtrace: 2020-10-27T04:08:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.880210", , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read" , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.851339", Ignoring malparsed backtrace: 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.948591", , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.545996", , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered" , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.961007", Ignoring malparsed backtrace: 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.570838", , 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched" , 2020-10-27T04:08:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.910208", Ignoring malparsed backtrace: 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.576005", , 2020-10-27T04:08:53.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.882882", Ignoring malparsed backtrace: 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.903080", , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" Ignoring malparsed backtrace: 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 645 , 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.822 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.026680145999999998, , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2020-10-27T04:08:53.862 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.891062", , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:50.383624", Ignoring malparsed backtrace: 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5225 , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:09.403819", , 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:53.931 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", Ignoring malparsed backtrace: 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.479655", Ignoring malparsed backtrace: 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.480291", , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:54.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.854040", Ignoring malparsed backtrace: 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.399499", , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.938560", Ignoring malparsed backtrace: 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.404750", , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.409309", , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered" , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4772 Ignoring malparsed backtrace: 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:29.380947", , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:37.399577", , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched" , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:11.433446", Ignoring malparsed backtrace: 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.511196", , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ Ignoring malparsed backtrace: 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.866526", , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.888348", Ignoring malparsed backtrace: 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.918309", , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.992 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917155", , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4570 Ignoring malparsed backtrace: 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:18.059358", , 2020-10-27T04:08:55.035 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:32.434336", , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ , 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.076 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:23.573555", Ignoring malparsed backtrace: 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:34.744493", , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.823720", , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917821", Ignoring malparsed backtrace: 2020-10-27T04:08:55.516 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.883656", , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:46.509124", , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done" , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.559 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.675162", , 2020-10-27T04:08:55.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.720798", , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.615 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.691906", , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5449 , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:09:13.013 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":142,"fsid":"4c43081c-176b-4fed-8aad-53b3ee89c837","created":"2020-10-27 04:06:32.593739","modified":"2020-10-27 04:09:12.969268","last_up_change":"2020-10-27 04:06:41.641767","last_in_change":"2020-10-27 04:07:25.367025","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-10-27 04:06:45.894327","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"90","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2020-10-27 04:07:02.815119","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":99,"snap_epoch":124,"pool_snaps":[],"removed_snaps":"[1~2d,2f~a,3a~2,3d~4,45~4,4a~7,53~5,59~6]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1df688f3-1d4d-4143-afcb-b4db528993bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":14262},{"type":"v1","addr":"172.21.15.92:6825","nonce":14262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":14262},{"type":"v1","addr":"172.21.15.92:6827","nonce":14262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":14262},{"type":"v1","addr":"172.21.15.92:6831","nonce":14262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":14262},{"type":"v1","addr":"172.21.15.92:6829","nonce":14262}]},"public_addr":"172.21.15.92:6825/14262","cluster_addr":"172.21.15.92:6827/14262","heartbeat_back_addr":"172.21.15.92:6831/14262","heartbeat_front_addr":"172.21.15.92:6829/14262","state":["exists","up"]},{"osd":1,"uuid":"d3f3dd2b-1944-4e9a-b751-f370a9f34f56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":14263},{"type":"v1","addr":"172.21.15.92:6810","nonce":14263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":14263},{"type":"v1","addr":"172.21.15.92:6814","nonce":14263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":14263},{"type":"v1","addr":"172.21.15.92:6823","nonce":14263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":14263},{"type":"v1","addr":"172.21.15.92:6819","nonce":14263}]},"public_addr":"172.21.15.92:6810/14263","cluster_addr":"172.21.15.92:6814/14263","heartbeat_back_addr":"172.21.15.92:6823/14263","heartbeat_front_addr":"172.21.15.92:6819/14263","state":["exists","up"]},{"osd":2,"uuid":"f13ee95a-f9be-4695-a412-e444a664d608","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":14264},{"type":"v1","addr":"172.21.15.92:6811","nonce":14264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":14264},{"type":"v1","addr":"172.21.15.92:6815","nonce":14264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":14264},{"type":"v1","addr":"172.21.15.92:6822","nonce":14264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":14264},{"type":"v1","addr":"172.21.15.92:6818","nonce":14264}]},"public_addr":"172.21.15.92:6811/14264","cluster_addr":"172.21.15.92:6815/14264","heartbeat_back_addr":"172.21.15.92:6822/14264","heartbeat_front_addr":"172.21.15.92:6818/14264","state":["exists","up"]},{"osd":3,"uuid":"6fbdb1f1-829a-4bf1-a5c4-2973d47a9fb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":14261},{"type":"v1","addr":"172.21.15.92:6801","nonce":14261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":14261},{"type":"v1","addr":"172.21.15.92:6803","nonce":14261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":14261},{"type":"v1","addr":"172.21.15.92:6807","nonce":14261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":14261},{"type":"v1","addr":"172.21.15.92:6805","nonce":14261}]},"public_addr":"172.21.15.92:6801/14261","cluster_addr":"172.21.15.92:6803/14261","heartbeat_back_addr":"172.21.15.92:6807/14261","heartbeat_front_addr":"172.21.15.92:6805/14261","state":["exists","up"]},{"osd":4,"uuid":"b88aa06a-d50b-497a-b8bb-578f3bd7585a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":14407},{"type":"v1","addr":"172.21.15.143:6819","nonce":14407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":14407},{"type":"v1","addr":"172.21.15.143:6821","nonce":14407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":14407},{"type":"v1","addr":"172.21.15.143:6825","nonce":14407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":14407},{"type":"v1","addr":"172.21.15.143:6823","nonce":14407}]},"public_addr":"172.21.15.143:6819/14407","cluster_addr":"172.21.15.143:6821/14407","heartbeat_back_addr":"172.21.15.143:6825/14407","heartbeat_front_addr":"172.21.15.143:6823/14407","state":["exists","up"]},{"osd":5,"uuid":"70ee976a-1c73-4a16-9772-ccb0b29ce83d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":14431},{"type":"v1","addr":"172.21.15.143:6827","nonce":14431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":14431},{"type":"v1","addr":"172.21.15.143:6829","nonce":14431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":14431},{"type":"v1","addr":"172.21.15.143:6833","nonce":14431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":14431},{"type":"v1","addr":"172.21.15.143:6831","nonce":14431}]},"public_addr":"172.21.15.143:6827/14431","cluster_addr":"172.21.15.143:6829/14431","heartbeat_back_addr":"172.21.15.143:6833/14431","heartbeat_front_addr":"172.21.15.143:6831/14431","state":["exists","up"]},{"osd":6,"uuid":"b5af2580-ae4b-4740-b443-d5e345375840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":14416},{"type":"v1","addr":"172.21.15.143:6811","nonce":14416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":14416},{"type":"v1","addr":"172.21.15.143:6813","nonce":14416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":14416},{"type":"v1","addr":"172.21.15.143:6817","nonce":14416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":14416},{"type":"v1","addr":"172.21.15.143:6815","nonce":14416}]},"public_addr":"172.21.15.143:6811/14416","cluster_addr":"172.21.15.143:6813/14416","heartbeat_back_addr":"172.21.15.143:6817/14416","heartbeat_front_addr":"172.21.15.143:6815/14416","state":["exists","up"]},{"osd":7,"uuid":"6c19afa9-e887-422e-8af6-bd96527de403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":14408},{"type":"v1","addr":"172.21.15.143:6803","nonce":14408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":14408},{"type":"v1","addr":"172.21.15.143:6805","nonce":14408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":14408},{"type":"v1","addr":"172.21.15.143:6809","nonce":14408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":14408},{"type":"v1","addr":"172.21.15.143:6807","nonce":14408}]},"public_addr":"172.21.15.143:6803/14408","cluster_addr":"172.21.15.143:6805/14408","heartbeat_back_addr":"172.21.15.143:6809/14408","heartbeat_front_addr":"172.21.15.143:6807/14408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":24,"length":1},{"begin":36,"length":1},{"begin":39,"length":1},{"begin":42,"length":1},{"begin":48,"length":3},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":69,"length":1},{"begin":72,"length":1},{"begin":74,"length":1},{"begin":77,"length":3},{"begin":85,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} Ignoring malparsed backtrace: 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ , 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v2", Ignoring malparsed backtrace: 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", Ignoring malparsed backtrace: 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "upmap_max_iterations", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.009 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_USER_ID": { , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_USER_ID", Ignoring malparsed backtrace: 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "deepsea", , 2020-10-27T04:09:20.067 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "device_health_metrics", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.095 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "diskprediction_default_authority", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "43200", , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "name prefix for any created local pool", , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, , 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "error_string": "", , 2020-10-27T04:09:20.169 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.198 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.226 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: "channel_crash": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "interval", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "identifier": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "identifier", Ignoring malparsed backtrace: 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "smtp_destination", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "beginning time of day to automatically balance", Ignoring malparsed backtrace: 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "secs", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "FEATURE_TOGGLE_mirroring": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "GRAFANA_API_USERNAME", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.385 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Enable/disable debug options", Ignoring malparsed backtrace: 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.414 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "self_heal": { , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "self_heal", Ignoring malparsed backtrace: 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.444 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "password", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "failure_domain": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "failure_domain", Ignoring malparsed backtrace: 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "rwoption5": { , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "rwoption5", Ignoring malparsed backtrace: 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: Set osd_debug_deep_scrub_sleep to 0 , 2020-10-27T04:09:25.675 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:10:07.915 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:07:57.387753 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.189 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:50.069 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.11 , instructing pg 1.11 on osd.6 to deep-scrub , 2020-10-27T04:11:50.484 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 Ignoring malparsed backtrace: 2020-10-27T04:12:22.404 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:13:23.752 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 2.cs0 on osd.6 to deep-scrub , 2020-10-27T04:14:07.859 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.8 , instructing pg 1.8 on osd.1 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:14:56.987 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.345 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:16:25.121 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.6 , instructing pg 2.6s0 on osd.4 to deep-scrub , 2020-10-27T04:16:25.501 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.4 Ignoring malparsed backtrace: 2020-10-27T04:17:11.077 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:17:11.078 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:17:31.079 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:18:12.385 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 1.1 on osd.3 to deep-scrub , 2020-10-27T04:18:42.606 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.2 , instructing pg 2.2s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:19:45.623 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.624 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.625 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.915 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:18.628 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:22:19.518 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , 2020-10-27T04:22:19.899 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all Ignoring malparsed backtrace: 2020-10-27T04:23:00.748 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.378 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:07:56.847884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.362 DEBUG:tasks.ceph.osd.4:waiting for process to exit , 2020-10-27T04:24:54.362 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:24:54.428 INFO:tasks.ceph.osd.4:Stopped Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f0c1f6ed652] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f0c1f6ed82d] Ignoring malparsed backtrace: 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.263 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.856351", Ignoring malparsed backtrace: 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.277 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:28.410791", , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" , 2020-10-27T04:08:52.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:11.431850", , 2020-10-27T04:08:52.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:52.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:28.593287", Ignoring malparsed backtrace: 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.334 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:37.693947", , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" , 2020-10-27T04:08:52.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.717939", , 2020-10-27T04:08:52.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:08:43.729941", Ignoring malparsed backtrace: 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.392 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.854204", Ignoring malparsed backtrace: 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:52.744 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.969023", , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:52.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:52.772 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:12.972324", , 2020-10-27T04:08:52.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:52.786 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:52.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:23.390225", Ignoring malparsed backtrace: 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.800 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:52.801 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.814 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", , 2020-10-27T04:08:52.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0098285809999999994, , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:52.832 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", Ignoring malparsed backtrace: 2020-10-27T04:08:52.853 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 41.220027107999996, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087482700000000007, , 2020-10-27T04:08:52.854 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { Ignoring malparsed backtrace: 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2020-10-27 04:08:34.643724", , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 18.088702560000002, , 2020-10-27T04:08:52.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074977419999999999, Ignoring malparsed backtrace: 2020-10-27T04:08:52.899 INFO:teuthology.orchestra.run.smithi092.stdout:} , 2020-10-27T04:08:52.900 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight , nodeep-scrub is set Ignoring malparsed backtrace: 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.202 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ , 2020-10-27T04:08:53.203 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.316031", Ignoring malparsed backtrace: 2020-10-27T04:08:53.230 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.880210", , 2020-10-27T04:08:53.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.244 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read" , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.258 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.851339", Ignoring malparsed backtrace: 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.948591", , 2020-10-27T04:08:53.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.545996", , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" , 2020-10-27T04:08:53.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered" , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.313 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.961007", Ignoring malparsed backtrace: 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", , 2020-10-27T04:08:53.340 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.570838", , 2020-10-27T04:08:53.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched" , 2020-10-27T04:08:53.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.367 INFO:teuthology.orchestra.run.smithi092.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.910208", Ignoring malparsed backtrace: 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:08:53.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2020-10-27 04:07:08.576005", , 2020-10-27T04:08:53.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated" Ignoring malparsed backtrace: 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.794 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.882882", Ignoring malparsed backtrace: 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.903080", , 2020-10-27T04:08:53.808 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" Ignoring malparsed backtrace: 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 645 , 2020-10-27T04:08:53.821 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.822 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for rw locks" , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.835 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 0.026680145999999998, , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:53.849 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", Ignoring malparsed backtrace: 2020-10-27T04:08:53.862 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.891062", , 2020-10-27T04:08:53.863 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.876 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.890 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.903 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:50.383624", Ignoring malparsed backtrace: 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5225 , 2020-10-27T04:08:53.917 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:09.403819", , 2020-10-27T04:08:53.930 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:53.931 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:53.945 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", Ignoring malparsed backtrace: 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.959 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.479655", Ignoring malparsed backtrace: 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.480291", , 2020-10-27T04:08:53.972 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:54.380 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.381 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.854040", Ignoring malparsed backtrace: 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "type_data": { , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "flag_point": "commit sent; apply or cleanup", , 2020-10-27T04:08:54.395 INFO:teuthology.orchestra.run.smithi143.stdout: "client_info": { Ignoring malparsed backtrace: 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.399499", , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.411 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.426 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.439 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.440 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.938560", Ignoring malparsed backtrace: 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.404750", , 2020-10-27T04:08:54.454 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.409309", , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "waiting for peered" , 2020-10-27T04:08:54.468 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.482 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:54.496 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4772 Ignoring malparsed backtrace: 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:29.380947", , 2020-10-27T04:08:54.510 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:37.399577", , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "dispatched" , 2020-10-27T04:08:54.524 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.537 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.538 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.551 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.552 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:11.433446", Ignoring malparsed backtrace: 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:19.511196", , 2020-10-27T04:08:54.565 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "size": 20, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "duration": 600, , 2020-10-27T04:08:54.923 INFO:teuthology.orchestra.run.smithi143.stdout: "ops": [ Ignoring malparsed backtrace: 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.866526", , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:54.937 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.951 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.965 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.888348", Ignoring malparsed backtrace: 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.918309", , 2020-10-27T04:08:54.979 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:08:54.992 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917155", , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:54.993 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.007 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client": "client.4531", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "client_addr": "172.21.15.92:0/485396265", , 2020-10-27T04:08:55.021 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 4570 Ignoring malparsed backtrace: 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.034 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:18.059358", , 2020-10-27T04:08:55.035 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:32.434336", , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.048 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "sub_op_started" , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.062 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ , 2020-10-27T04:08:55.075 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.076 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:23.573555", Ignoring malparsed backtrace: 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:34.744493", , 2020-10-27T04:08:55.089 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "commit_sent" Ignoring malparsed backtrace: 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.103 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.473 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.823720", , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "started" , 2020-10-27T04:08:55.474 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "initiated" , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.489 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.503 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.917821", Ignoring malparsed backtrace: 2020-10-27T04:08:55.516 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:08.883656", , 2020-10-27T04:08:55.517 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" Ignoring malparsed backtrace: 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:07:46.509124", , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "done" , 2020-10-27T04:08:55.531 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.546 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.559 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.560 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.574 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.675162", , 2020-10-27T04:08:55.575 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "all_read" Ignoring malparsed backtrace: 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.720798", , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "reached_pg" , 2020-10-27T04:08:55.588 INFO:teuthology.orchestra.run.smithi143.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "throttled" , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.602 INFO:teuthology.orchestra.run.smithi143.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:08:55.615 INFO:teuthology.orchestra.run.smithi143.stdout: } , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: ] , 2020-10-27T04:08:55.616 INFO:teuthology.orchestra.run.smithi143.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: { , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "time": "2020-10-27 04:08:01.691906", , 2020-10-27T04:08:55.629 INFO:teuthology.orchestra.run.smithi143.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "tid": 5449 , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: }, , 2020-10-27T04:08:55.643 INFO:teuthology.orchestra.run.smithi143.stdout: "events": [ Ignoring malparsed backtrace: 2020-10-27T04:09:13.013 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd dump --format=json , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: , 2020-10-27T04:09:13.427 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":142,"fsid":"4c43081c-176b-4fed-8aad-53b3ee89c837","created":"2020-10-27 04:06:32.593739","modified":"2020-10-27 04:09:12.969268","last_up_change":"2020-10-27 04:06:41.641767","last_in_change":"2020-10-27 04:07:25.367025","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"nautilus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-10-27 04:06:45.894327","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"90","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2020-10-27 04:07:02.815119","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":3,"min_size":2,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"warn","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":99,"snap_epoch":124,"pool_snaps":[],"removed_snaps":"[1~2d,2f~a,3a~2,3d~4,45~4,4a~7,53~5,59~6]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"teuthologyprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"1df688f3-1d4d-4143-afcb-b4db528993bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":14262},{"type":"v1","addr":"172.21.15.92:6825","nonce":14262}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":14262},{"type":"v1","addr":"172.21.15.92:6827","nonce":14262}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":14262},{"type":"v1","addr":"172.21.15.92:6831","nonce":14262}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":14262},{"type":"v1","addr":"172.21.15.92:6829","nonce":14262}]},"public_addr":"172.21.15.92:6825/14262","cluster_addr":"172.21.15.92:6827/14262","heartbeat_back_addr":"172.21.15.92:6831/14262","heartbeat_front_addr":"172.21.15.92:6829/14262","state":["exists","up"]},{"osd":1,"uuid":"d3f3dd2b-1944-4e9a-b751-f370a9f34f56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":130,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":14263},{"type":"v1","addr":"172.21.15.92:6810","nonce":14263}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":14263},{"type":"v1","addr":"172.21.15.92:6814","nonce":14263}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6821","nonce":14263},{"type":"v1","addr":"172.21.15.92:6823","nonce":14263}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6817","nonce":14263},{"type":"v1","addr":"172.21.15.92:6819","nonce":14263}]},"public_addr":"172.21.15.92:6810/14263","cluster_addr":"172.21.15.92:6814/14263","heartbeat_back_addr":"172.21.15.92:6823/14263","heartbeat_front_addr":"172.21.15.92:6819/14263","state":["exists","up"]},{"osd":2,"uuid":"f13ee95a-f9be-4695-a412-e444a664d608","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6809","nonce":14264},{"type":"v1","addr":"172.21.15.92:6811","nonce":14264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6813","nonce":14264},{"type":"v1","addr":"172.21.15.92:6815","nonce":14264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":14264},{"type":"v1","addr":"172.21.15.92:6822","nonce":14264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":14264},{"type":"v1","addr":"172.21.15.92:6818","nonce":14264}]},"public_addr":"172.21.15.92:6811/14264","cluster_addr":"172.21.15.92:6815/14264","heartbeat_back_addr":"172.21.15.92:6822/14264","heartbeat_front_addr":"172.21.15.92:6818/14264","state":["exists","up"]},{"osd":3,"uuid":"6fbdb1f1-829a-4bf1-a5c4-2973d47a9fb5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":14261},{"type":"v1","addr":"172.21.15.92:6801","nonce":14261}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":14261},{"type":"v1","addr":"172.21.15.92:6803","nonce":14261}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":14261},{"type":"v1","addr":"172.21.15.92:6807","nonce":14261}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":14261},{"type":"v1","addr":"172.21.15.92:6805","nonce":14261}]},"public_addr":"172.21.15.92:6801/14261","cluster_addr":"172.21.15.92:6803/14261","heartbeat_back_addr":"172.21.15.92:6807/14261","heartbeat_front_addr":"172.21.15.92:6805/14261","state":["exists","up"]},{"osd":4,"uuid":"b88aa06a-d50b-497a-b8bb-578f3bd7585a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":14407},{"type":"v1","addr":"172.21.15.143:6819","nonce":14407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":14407},{"type":"v1","addr":"172.21.15.143:6821","nonce":14407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6824","nonce":14407},{"type":"v1","addr":"172.21.15.143:6825","nonce":14407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":14407},{"type":"v1","addr":"172.21.15.143:6823","nonce":14407}]},"public_addr":"172.21.15.143:6819/14407","cluster_addr":"172.21.15.143:6821/14407","heartbeat_back_addr":"172.21.15.143:6825/14407","heartbeat_front_addr":"172.21.15.143:6823/14407","state":["exists","up"]},{"osd":5,"uuid":"70ee976a-1c73-4a16-9772-ccb0b29ce83d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6826","nonce":14431},{"type":"v1","addr":"172.21.15.143:6827","nonce":14431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6828","nonce":14431},{"type":"v1","addr":"172.21.15.143:6829","nonce":14431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6832","nonce":14431},{"type":"v1","addr":"172.21.15.143:6833","nonce":14431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6830","nonce":14431},{"type":"v1","addr":"172.21.15.143:6831","nonce":14431}]},"public_addr":"172.21.15.143:6827/14431","cluster_addr":"172.21.15.143:6829/14431","heartbeat_back_addr":"172.21.15.143:6833/14431","heartbeat_front_addr":"172.21.15.143:6831/14431","state":["exists","up"]},{"osd":6,"uuid":"b5af2580-ae4b-4740-b443-d5e345375840","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":14416},{"type":"v1","addr":"172.21.15.143:6811","nonce":14416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":14416},{"type":"v1","addr":"172.21.15.143:6813","nonce":14416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":14416},{"type":"v1","addr":"172.21.15.143:6817","nonce":14416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":14416},{"type":"v1","addr":"172.21.15.143:6815","nonce":14416}]},"public_addr":"172.21.15.143:6811/14416","cluster_addr":"172.21.15.143:6813/14416","heartbeat_back_addr":"172.21.15.143:6817/14416","heartbeat_front_addr":"172.21.15.143:6815/14416","state":["exists","up"]},{"osd":7,"uuid":"6c19afa9-e887-422e-8af6-bd96527de403","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":14408},{"type":"v1","addr":"172.21.15.143:6803","nonce":14408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":14408},{"type":"v1","addr":"172.21.15.143:6805","nonce":14408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":14408},{"type":"v1","addr":"172.21.15.143:6809","nonce":14408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":14408},{"type":"v1","addr":"172.21.15.143:6807","nonce":14408}]},"public_addr":"172.21.15.143:6803/14408","cluster_addr":"172.21.15.143:6805/14408","heartbeat_back_addr":"172.21.15.143:6809/14408","heartbeat_front_addr":"172.21.15.143:6807/14408","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4611087854031667199,"old_weight":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"teuthologyprofile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":24,"length":1},{"begin":36,"length":1},{"begin":39,"length":1},{"begin":42,"length":1},{"begin":48,"length":3},{"begin":54,"length":1},{"begin":56,"length":1},{"begin":59,"length":1},{"begin":61,"length":2},{"begin":69,"length":1},{"begin":72,"length":1},{"begin":74,"length":1},{"begin":77,"length":3},{"begin":85,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} Ignoring malparsed backtrace: 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ , 2020-10-27T04:09:19.907 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.908 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v2", Ignoring malparsed backtrace: 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:19.921 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", Ignoring malparsed backtrace: 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:19.935 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:19.950 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:19.964 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:19.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "upmap_max_iterations", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "uint", , 2020-10-27T04:09:19.979 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "True", , 2020-10-27T04:09:19.994 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.008 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.009 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.023 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "RGW_API_USER_ID": { , 2020-10-27T04:09:20.038 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "RGW_API_USER_ID", Ignoring malparsed backtrace: 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: { , 2020-10-27T04:09:20.066 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "deepsea", , 2020-10-27T04:09:20.067 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "device_health_metrics", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.081 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.095 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "diskprediction_default_authority", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.096 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "43200", , 2020-10-27T04:09:20.110 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.125 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.139 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "name prefix for any created local pool", , 2020-10-27T04:09:20.154 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "can_run": true, , 2020-10-27T04:09:20.168 INFO:teuthology.orchestra.run.smithi092.stdout: "error_string": "", , 2020-10-27T04:09:20.169 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "key_file", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.183 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.197 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.198 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.226 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.227 INFO:teuthology.orchestra.run.smithi092.stdout: "channel_crash": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "interval", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "int", , 2020-10-27T04:09:20.241 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.255 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "identifier": { , 2020-10-27T04:09:20.256 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "identifier", Ignoring malparsed backtrace: 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "smtp_destination", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.270 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.284 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.299 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "beginning time of day to automatically balance", Ignoring malparsed backtrace: 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.313 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "secs", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.328 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, Ignoring malparsed backtrace: 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.342 INFO:teuthology.orchestra.run.smithi092.stdout: "FEATURE_TOGGLE_mirroring": { Ignoring malparsed backtrace: 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "GRAFANA_API_USERNAME", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.357 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.371 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.385 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.386 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "Enable/disable debug options", Ignoring malparsed backtrace: 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], , 2020-10-27T04:09:20.400 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:09:20.414 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 0, , 2020-10-27T04:09:20.415 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "self_heal": { , 2020-10-27T04:09:20.429 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "self_heal", Ignoring malparsed backtrace: 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.443 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.444 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: "see_also": [] , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } , 2020-10-27T04:09:20.458 INFO:teuthology.orchestra.run.smithi092.stdout: } Ignoring malparsed backtrace: 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "password", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "str", , 2020-10-27T04:09:20.472 INFO:teuthology.orchestra.run.smithi092.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "module_options": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "failure_domain": { , 2020-10-27T04:09:20.487 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "failure_domain", Ignoring malparsed backtrace: 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.501 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.502 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "default_value": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.516 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.530 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "rwoption5": { , 2020-10-27T04:09:20.545 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "rwoption5", Ignoring malparsed backtrace: 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.559 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.573 INFO:teuthology.orchestra.run.smithi092.stdout: "min": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.574 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "long_desc": "", , 2020-10-27T04:09:20.588 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "max": "", , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "enum_allowed": [], , 2020-10-27T04:09:20.603 INFO:teuthology.orchestra.run.smithi092.stdout: "desc": "", Ignoring malparsed backtrace: Set osd_debug_deep_scrub_sleep to 0 , 2020-10-27T04:09:25.675 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 5 , instructed osd(s) 5 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:10:07.915 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:07.916 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:07:57.387753 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.188 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:09.189 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:11:50.069 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.11 , instructing pg 1.11 on osd.6 to deep-scrub , 2020-10-27T04:11:50.484 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.10 Ignoring malparsed backtrace: 2020-10-27T04:12:22.404 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:22.405 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:13:23.752 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:13:23.753 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 2.cs0 on osd.6 to deep-scrub , 2020-10-27T04:14:07.859 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.8 , instructing pg 1.8 on osd.1 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:14:56.987 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:14:56.988 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.344 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:58.345 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:16:25.121 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.6 , instructing pg 2.6s0 on osd.4 to deep-scrub , 2020-10-27T04:16:25.501 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.4 Ignoring malparsed backtrace: 2020-10-27T04:17:11.077 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:17:11.078 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:17:31.079 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:18:12.385 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:18:12.386 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 1.1 on osd.3 to deep-scrub , 2020-10-27T04:18:42.606 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.2 , instructing pg 2.2s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:19:45.623 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.624 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:19:45.625 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 1.e last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.914 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:46.915 INFO:tasks.ceph:pgid 1.f last_scrub_stamp 2020-10-27 04:08:08.329963 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=8, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:18.628 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:07:56.366734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:18.629 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:22:19.518 INFO:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , 2020-10-27T04:22:19.899 INFO:teuthology.orchestra.run.smithi092.stdout: , dumped all Ignoring malparsed backtrace: 2020-10-27T04:23:00.748 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:00.749 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:06:46.596884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=6, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.377 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:08:07.425973 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=8, tm_sec=7, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:32.378 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:07:56.847884 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:33.620 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:07:03.746412 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=7, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=9, tm_sec=22, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.362 DEBUG:tasks.ceph.osd.4:waiting for process to exit , 2020-10-27T04:24:54.362 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:24:54.428 INFO:tasks.ceph.osd.4:Stopped Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 47, in reraise , raise value.with_traceback(tb) Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fb9c08674f4] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x521) [0x5604e48be0b1] Ignoring malparsed backtrace: 2020-10-27T03:59:31.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:57:37.579170", , 2020-10-27T03:59:31.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" , 2020-10-27T03:59:31.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:31.973 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:31.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:57:37.586152", , 2020-10-27T03:59:31.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" Ignoring malparsed backtrace: 2020-10-27T03:59:31.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:31.992 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:31.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:57:37.612216", Ignoring malparsed backtrace: 2020-10-27T03:59:32.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read" , 2020-10-27T03:59:32.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.009 INFO:teuthology.orchestra.run.smithi191.stdout: { Ignoring malparsed backtrace: 2020-10-27T03:59:32.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:00.680828", , 2020-10-27T03:59:32.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg" , 2020-10-27T03:59:32.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.047 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:23.824689", , 2020-10-27T03:59:32.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T03:59:32.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.061 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:27.894257", Ignoring malparsed backtrace: 2020-10-27T03:59:32.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" , 2020-10-27T03:59:32.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.075 INFO:teuthology.orchestra.run.smithi191.stdout: { Ignoring malparsed backtrace: 2020-10-27T03:59:32.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:28.701997", , 2020-10-27T03:59:32.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" , 2020-10-27T03:59:32.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.103 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:59:17.567090", , 2020-10-27T03:59:32.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" Ignoring malparsed backtrace: 2020-10-27T03:59:32.747 INFO:teuthology.orchestra.run.smithi191.stdout:{ , 2020-10-27T03:59:32.748 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, , 2020-10-27T03:59:32.748 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, Ignoring malparsed backtrace: 2020-10-27T03:59:32.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:07.371857", , 2020-10-27T03:59:32.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_started" , 2020-10-27T03:59:32.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.776 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:27.852579", Ignoring malparsed backtrace: 2020-10-27T03:59:32.789 INFO:teuthology.orchestra.run.smithi191.stdout: ] , 2020-10-27T03:59:32.789 INFO:teuthology.orchestra.run.smithi191.stdout: } , 2020-10-27T03:59:32.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:27.890036", , 2020-10-27T03:59:32.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched" , 2020-10-27T03:59:32.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started" , 2020-10-27T03:59:32.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.817 INFO:teuthology.orchestra.run.smithi191.stdout: { Ignoring malparsed backtrace: 2020-10-27T03:59:32.830 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:58:52.844961", , 2020-10-27T03:59:32.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started" Ignoring malparsed backtrace: 2020-10-27T03:59:32.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg" , 2020-10-27T03:59:32.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.844 INFO:teuthology.orchestra.run.smithi191.stdout: { Ignoring malparsed backtrace: 2020-10-27T03:59:32.858 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:59:08.128616", , 2020-10-27T03:59:32.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg" Ignoring malparsed backtrace: 2020-10-27T03:59:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:59:08.654948", , 2020-10-27T03:59:32.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent" , 2020-10-27T03:59:32.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, Ignoring malparsed backtrace: 2020-10-27T03:59:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, , 2020-10-27T03:59:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:59:08.649851", Ignoring malparsed backtrace: 2020-10-27T03:59:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: { , 2020-10-27T03:59:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2020-10-27 03:59:22.246201", , 2020-10-27T03:59:32.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read" Ignoring malparsed backtrace: 2020-10-27T03:59:37.644 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 , 2020-10-27T03:59:37.644 INFO:teuthology.orchestra.run.smithi191.stdout:} , 2020-10-27T03:59:37.654 INFO:tasks.thrashosds.thrasher:inning osd Ignoring malparsed backtrace: 2020-10-27T04:00:07.095 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete , 2020-10-27T04:00:07.096 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , 2020-10-27T04:00:07.603 INFO:teuthology.orchestra.run.smithi136.stdout: Ignoring malparsed backtrace: 2020-10-27T04:00:08.873 INFO:teuthology.orchestra.run.smithi136.stdout: } , 2020-10-27T04:00:08.873 INFO:teuthology.orchestra.run.smithi136.stdout: ] , 2020-10-27T04:00:08.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:00:08.887 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "smtp_from_name", , 2020-10-27T04:00:08.887 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", , 2020-10-27T04:00:08.888 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, , 2020-10-27T04:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "error_string": "", , 2020-10-27T04:00:08.901 INFO:teuthology.orchestra.run.smithi136.stdout: "module_options": { Ignoring malparsed backtrace: 2020-10-27T04:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "7", , 2020-10-27T04:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], , 2020-10-27T04:00:08.915 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "Restrict automatic balancing to this day of the week or later", Ignoring malparsed backtrace: 2020-10-27T04:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "none", , 2020-10-27T04:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: "upmap" , 2020-10-27T04:00:08.929 INFO:teuthology.orchestra.run.smithi136.stdout: ], Ignoring malparsed backtrace: 2020-10-27T04:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: { , 2020-10-27T04:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "dashboard", , 2020-10-27T04:00:08.944 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: "see_also": [] , 2020-10-27T04:00:08.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:08.959 INFO:teuthology.orchestra.run.smithi136.stdout: "FEATURE_TOGGLE_rbd": { Ignoring malparsed backtrace: 2020-10-27T04:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", , 2020-10-27T04:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "bool", , 2020-10-27T04:00:08.973 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, , 2020-10-27T04:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "80", , 2020-10-27T04:00:08.987 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], , 2020-10-27T04:00:09.005 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: "redirect" , 2020-10-27T04:00:09.021 INFO:teuthology.orchestra.run.smithi136.stdout: ], , 2020-10-27T04:00:09.022 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.043 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", , 2020-10-27T04:00:09.043 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, , 2020-10-27T04:00:09.043 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:09.066 INFO:teuthology.orchestra.run.smithi136.stdout: "sleep_interval": { , 2020-10-27T04:00:09.066 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "sleep_interval", Ignoring malparsed backtrace: 2020-10-27T04:00:09.095 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", , 2020-10-27T04:00:09.095 INFO:teuthology.orchestra.run.smithi136.stdout: "long_desc": "", , 2020-10-27T04:00:09.095 INFO:teuthology.orchestra.run.smithi136.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.115 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "86400", , 2020-10-27T04:00:09.115 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.115 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.132 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "port", , 2020-10-27T04:00:09.132 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", , 2020-10-27T04:00:09.132 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:09.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:09.149 INFO:teuthology.orchestra.run.smithi136.stdout: "min_size": { , 2020-10-27T04:00:09.149 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "min_size", Ignoring malparsed backtrace: 2020-10-27T04:00:09.163 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", , 2020-10-27T04:00:09.163 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", , 2020-10-27T04:00:09.163 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:00:09.178 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "", , 2020-10-27T04:00:09.178 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.179 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.193 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "selftest", , 2020-10-27T04:00:09.193 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, , 2020-10-27T04:00:09.193 INFO:teuthology.orchestra.run.smithi136.stdout: "error_string": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:09.207 INFO:teuthology.orchestra.run.smithi136.stdout: "rwoption6": { , 2020-10-27T04:00:09.207 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "rwoption6", Ignoring malparsed backtrace: 2020-10-27T04:00:09.221 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "telegraf", , 2020-10-27T04:00:09.221 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, , 2020-10-27T04:00:09.221 INFO:teuthology.orchestra.run.smithi136.stdout: "error_string": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.235 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.235 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.235 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.249 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", , 2020-10-27T04:00:09.249 INFO:teuthology.orchestra.run.smithi136.stdout: "long_desc": "", , 2020-10-27T04:00:09.249 INFO:teuthology.orchestra.run.smithi136.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.263 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.263 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], , 2020-10-27T04:00:09.263 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.277 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "Password to authenticate with", , 2020-10-27T04:00:09.277 INFO:teuthology.orchestra.run.smithi136.stdout: "long_desc": "", , 2020-10-27T04:00:09.277 INFO:teuthology.orchestra.run.smithi136.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.291 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "", , 2020-10-27T04:00:09.292 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.292 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.306 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "crush_compat_step", , 2020-10-27T04:00:09.306 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "float", , 2020-10-27T04:00:09.306 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:09.320 INFO:teuthology.orchestra.run.smithi136.stdout: "upmap_max_deviation": { , 2020-10-27T04:00:09.320 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "upmap_max_deviation", , 2020-10-27T04:00:09.320 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:00:09.334 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", , 2020-10-27T04:00:09.334 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, , 2020-10-27T04:00:09.334 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "True", Ignoring malparsed backtrace: 2020-10-27T04:00:09.348 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.348 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.348 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.362 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", , 2020-10-27T04:00:09.362 INFO:teuthology.orchestra.run.smithi136.stdout: "long_desc": "", , 2020-10-27T04:00:09.363 INFO:teuthology.orchestra.run.smithi136.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.376 INFO:teuthology.orchestra.run.smithi136.stdout: "see_also": [] , 2020-10-27T04:00:09.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:09.376 INFO:teuthology.orchestra.run.smithi136.stdout: "RGW_API_SSL_VERIFY": { Ignoring malparsed backtrace: 2020-10-27T04:00:09.390 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "server_addr", , 2020-10-27T04:00:09.390 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", , 2020-10-27T04:00:09.390 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:09.404 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "username", , 2020-10-27T04:00:09.404 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", , 2020-10-27T04:00:09.404 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", Ignoring malparsed backtrace: 2020-10-27T04:00:09.418 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 1, , 2020-10-27T04:00:09.418 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "2419200", , 2020-10-27T04:00:09.418 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.432 INFO:teuthology.orchestra.run.smithi136.stdout: "diskprediction_cert_context": { , 2020-10-27T04:00:09.432 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "diskprediction_cert_context", , 2020-10-27T04:00:09.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "str", Ignoring malparsed backtrace: 2020-10-27T04:00:09.447 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", , 2020-10-27T04:00:09.448 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, , 2020-10-27T04:00:09.448 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "600", Ignoring malparsed backtrace: 2020-10-27T04:00:09.463 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", , 2020-10-27T04:00:09.463 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.463 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.477 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", , 2020-10-27T04:00:09.477 INFO:teuthology.orchestra.run.smithi136.stdout: "long_desc": "", , 2020-10-27T04:00:09.477 INFO:teuthology.orchestra.run.smithi136.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:00:09.491 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.491 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], , 2020-10-27T04:00:09.491 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "default pg_num for any created local pool", Ignoring malparsed backtrace: 2020-10-27T04:00:09.504 INFO:teuthology.orchestra.run.smithi136.stdout: "level": "advanced", , 2020-10-27T04:00:09.505 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 1, , 2020-10-27T04:00:09.505 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "5", Ignoring malparsed backtrace: 2020-10-27T04:00:09.518 INFO:teuthology.orchestra.run.smithi136.stdout: } , 2020-10-27T04:00:09.518 INFO:teuthology.orchestra.run.smithi136.stdout: } , 2020-10-27T04:00:09.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:00:09.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flags": 0, , 2020-10-27T04:00:09.532 INFO:teuthology.orchestra.run.smithi136.stdout: "default_value": "", , 2020-10-27T04:00:09.532 INFO:teuthology.orchestra.run.smithi136.stdout: "min": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.546 INFO:teuthology.orchestra.run.smithi136.stdout: "max": "", , 2020-10-27T04:00:09.546 INFO:teuthology.orchestra.run.smithi136.stdout: "enum_allowed": [], , 2020-10-27T04:00:09.546 INFO:teuthology.orchestra.run.smithi136.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.559 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, , 2020-10-27T04:00:09.559 INFO:teuthology.orchestra.run.smithi136.stdout: "error_string": "", , 2020-10-27T04:00:09.559 INFO:teuthology.orchestra.run.smithi136.stdout: "module_options": { Ignoring malparsed backtrace: 2020-10-27T04:00:09.573 INFO:teuthology.orchestra.run.smithi136.stdout: "enabled": { , 2020-10-27T04:00:09.573 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "enabled", , 2020-10-27T04:00:09.573 INFO:teuthology.orchestra.run.smithi136.stdout: "type": "bool", Ignoring malparsed backtrace: 2020-10-27T04:00:09.587 INFO:teuthology.orchestra.run.smithi136.stdout: "name": "test_orchestrator", , 2020-10-27T04:00:09.587 INFO:teuthology.orchestra.run.smithi136.stdout: "can_run": true, , 2020-10-27T04:00:09.587 INFO:teuthology.orchestra.run.smithi136.stdout: "error_string": "", Ignoring malparsed backtrace: 2020-10-27T04:00:09.600 INFO:teuthology.orchestra.run.smithi136.stdout: } , 2020-10-27T04:00:09.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, , 2020-10-27T04:00:09.601 INFO:teuthology.orchestra.run.smithi136.stdout: "servicemap": { Ignoring malparsed backtrace: 2020-10-27T04:00:43.234 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 03:58:33.354721 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=33, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:00:43.235 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:00:43.235 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-27 03:58:16.045136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=16, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:01:45.129 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:01:45.129 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:01:45.130 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. Ignoring malparsed backtrace: instructing pg 2.6s0 on osd.1 to deep-scrub , 2020-10-27T04:02:51.872 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.4 , instructing pg 1.4 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:04:03.416 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:04:03.416 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-10-27 03:57:09.775736 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=9, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:04:03.417 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: instructing pg 2.fs0 on osd.4 to deep-scrub , 2020-10-27T04:05:06.063 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.e , instructing pg 2.es0 on osd.2 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:06:01.357 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:06:01.357 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:06:01.357 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. Ignoring malparsed backtrace: 2020-10-27T04:07:24.057 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 03:58:28.315613 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=28, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:07:24.057 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:07:24.058 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 03:58:16.887674 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=16, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:08:20.255 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:08:20.255 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-10-27 03:57:09.775736 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=9, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:08:20.255 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:09:42.228 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all , 2020-10-27T04:09:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: Ignoring malparsed backtrace: 2020-10-27T04:10:17.737 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:17.737 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:10:17.737 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. Ignoring malparsed backtrace: 2020-10-27T04:11:40.362 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 03:58:28.315613 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=28, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:40.363 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:11:40.363 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 03:58:16.887674 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=16, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:12:35.851 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2020-10-27 03:58:16.045136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=16, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:35.851 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:12:35.852 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2020-10-27 03:57:09.775736 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=9, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:13:37.612 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:13:57.613 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all Ignoring malparsed backtrace: instructing pg 2.cs0 on osd.2 to deep-scrub , 2020-10-27T04:14:32.478 INFO:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.d , instructing pg 2.ds0 on osd.1 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:15:55.676 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:55.677 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 03:58:28.315613 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=58, tm_sec=28, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:15:55.677 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 03:57:34.523734 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=3, tm_min=57, tm_sec=34, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=0, tm_sec=12, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:16:16.677 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:16:16.717 INFO:tasks.ceph.osd.7:Stopped , 2020-10-27T04:16:16.718 INFO:teuthology.misc:Shutting down mgr daemons... Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 178, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 20 --max-in-flight 8 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 0 --op write 0 --op delete 20 --op append 5 --op write_excl 0 --op append_excl 5 --pool unique_pool_0' Ignoring malparsed backtrace: 2020-10-27T04:16:43.355 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2020-10-27T04:16:43.368 INFO:teuthology.task.internal:Transferring archived files... , 2020-10-27T04:16:43.368 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/xxg-2020-10-27_03:08:29-rados:thrash-erasure-code-nautilus-testing-1027-distro-basic-smithi/5563940/remote/smithi136 Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f173b3864f4] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1292) [0x55e87ba78562] Ignoring malparsed backtrace: 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:18.424908", , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:32.937349", , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.040020", , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled" , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done" , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: } , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] Ignoring malparsed backtrace: 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.044446", Ignoring malparsed backtrace: 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg" , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:44.024020", , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:53.982234", Ignoring malparsed backtrace: 2020-10-27T04:18:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:54.002520", , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4294", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.39:0/1113100290", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 673 Ignoring malparsed backtrace: 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:56.931035", , 2020-10-27T04:18:58.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:19:31.121 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 7 1 , reweighted osd.7 to 1 (10000) , 2020-10-27T04:19:32.180 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd primary-affinity 7 1 Ignoring malparsed backtrace: 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus" , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: ], Ignoring malparsed backtrace: 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "mgrmap": { , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 3, , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 4104, Ignoring malparsed backtrace: 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", Ignoring malparsed backtrace: 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", Ignoring malparsed backtrace: 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to persist completed events", , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": {} Ignoring malparsed backtrace: 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", Ignoring malparsed backtrace: 2020-10-27T04:19:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { , 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", , 2020-10-27T04:19:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", Ignoring malparsed backtrace: 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "diskprediction_port": { , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", , 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", Ignoring malparsed backtrace: 2020-10-27T04:19:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", Ignoring malparsed backtrace: 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", Ignoring malparsed backtrace: 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:54.239 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2020-10-27T04:19:54.899 INFO:tasks.ceph:Scrubbing osd.4 Ignoring malparsed backtrace: 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.10 last_scrub_stamp 2020-10-27 04:18:57.342688 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.270 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:22.647 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:22.647 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:21:42.649 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:22:24.804 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:07.878 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:18:54.119312 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:09.996 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.522 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.e , instructing pg 2.es0 on osd.4 to deep-scrub , 2020-10-27T04:24:55.189 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.7 Ignoring malparsed backtrace: 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:26:35.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:26:55.674 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all Ignoring malparsed backtrace: instructing pg 1.0 on osd.1 to deep-scrub , 2020-10-27T04:27:27.955 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.3 , instructing pg 2.3s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:59.786 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.a , instructing pg 1.a on osd.1 to deep-scrub , 2020-10-27T04:30:00.405 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.9 Ignoring malparsed backtrace: 2020-10-27T04:31:08.660 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: dumped all , 2020-10-27T04:32:10.678 INFO:teuthology.orchestra.run.smithi039.stdout: , 2020-10-27T04:32:10.701 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2020-10-27 04:32:09.052368","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8538736,"kb_used_data":150064,"kb_used_omap":7,"kb_used_meta":8388600,"kb_avail":746435984,"statfs":{"total":773094113280,"available":764350447616,"internally_reserved":8589934592,"allocated":153665536,"data_stored":180304914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":8589926562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.11","version":"0'0","reported_seq":"84","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.426833","last_change":"2020-10-27 04:18:55.156939","last_active":"2020-10-27 04:19:34.426833","last_peered":"2020-10-27 04:19:34.426833","last_clean":"2020-10-27 04:19:34.426833","last_became_active":"2020-10-27 04:18:53.912405","last_became_peered":"2020-10-27 04:18:53.912405","last_unstale":"2020-10-27 04:19:34.426833","last_undegraded":"2020-10-27 04:19:34.426833","last_fullsized":"2020-10-27 04:19:34.426833","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.156652","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.156652","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":"38","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602128","last_change":"2020-10-27 04:18:57.342812","last_active":"2020-10-27 04:19:34.602128","last_peered":"2020-10-27 04:19:34.602128","last_clean":"2020-10-27 04:19:34.602128","last_became_active":"2020-10-27 04:18:53.913747","last_became_peered":"2020-10-27 04:18:53.913747","last_unstale":"2020-10-27 04:19:34.602128","last_undegraded":"2020-10-27 04:19:34.602128","last_fullsized":"2020-10-27 04:19:34.602128","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.342688","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.342688","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"39","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427250","last_change":"2020-10-27 04:18:57.141096","last_active":"2020-10-27 04:19:34.427250","last_peered":"2020-10-27 04:19:34.427250","last_clean":"2020-10-27 04:19:34.427250","last_became_active":"2020-10-27 04:18:53.913837","last_became_peered":"2020-10-27 04:18:53.913837","last_unstale":"2020-10-27 04:19:34.427250","last_undegraded":"2020-10-27 04:19:34.427250","last_fullsized":"2020-10-27 04:19:34.427250","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.140975","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.140975","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.f","version":"63'137","reported_seq":"305","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602824","last_change":"2020-10-27 04:18:55.963308","last_active":"2020-10-27 04:19:34.602824","last_peered":"2020-10-27 04:19:34.602824","last_clean":"2020-10-27 04:19:34.602824","last_became_active":"2020-10-27 04:18:14.269110","last_became_peered":"2020-10-27 04:18:14.269110","last_unstale":"2020-10-27 04:19:34.602824","last_undegraded":"2020-10-27 04:19:34.602824","last_fullsized":"2020-10-27 04:19:34.602824","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13731319,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":5580,"num_write":76,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"95","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608614","last_change":"2020-10-27 04:18:56.256991","last_active":"2020-10-27 04:19:34.608614","last_peered":"2020-10-27 04:19:34.608614","last_clean":"2020-10-27 04:19:34.608614","last_became_active":"2020-10-27 04:18:53.917145","last_became_peered":"2020-10-27 04:18:53.917145","last_unstale":"2020-10-27 04:19:34.608614","last_undegraded":"2020-10-27 04:19:34.608614","last_fullsized":"2020-10-27 04:19:34.608614","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:56.256846","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:56.256846","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"55'8","reported_seq":"150","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142999","last_change":"2020-10-27 04:19:27.888673","last_active":"2020-10-27 04:19:34.142999","last_peered":"2020-10-27 04:19:34.142999","last_clean":"2020-10-27 04:19:34.142999","last_became_active":"2020-10-27 04:19:27.887042","last_became_peered":"2020-10-27 04:19:27.887042","last_unstale":"2020-10-27 04:19:34.142999","last_undegraded":"2020-10-27 04:19:34.142999","last_fullsized":"2020-10-27 04:19:34.142999","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":470596,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1382,"num_write":4,"num_write_kb":460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608889","last_change":"2020-10-27 04:18:58.356254","last_active":"2020-10-27 04:19:34.608889","last_peered":"2020-10-27 04:19:34.608889","last_clean":"2020-10-27 04:19:34.608889","last_became_active":"2020-10-27 04:18:45.994526","last_became_peered":"2020-10-27 04:18:45.994526","last_unstale":"2020-10-27 04:19:34.608889","last_undegraded":"2020-10-27 04:19:34.608889","last_fullsized":"2020-10-27 04:19:34.608889","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:58.356124","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:58.356124","last_clean_scrub_stamp":"2020-10-27 04:18:58.356124","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"65'41","reported_seq":"224","reported_epoch":"88","state":"active+clean+scrubbing","last_fresh":"2020-10-27 04:19:34.605028","last_change":"2020-10-27 04:19:14.461624","last_active":"2020-10-27 04:19:34.605028","last_peered":"2020-10-27 04:19:34.605028","last_clean":"2020-10-27 04:19:34.605028","last_became_active":"2020-10-27 04:18:14.276711","last_became_peered":"2020-10-27 04:18:14.276711","last_unstale":"2020-10-27 04:19:34.605028","last_undegraded":"2020-10-27 04:19:34.605028","last_fullsized":"2020-10-27 04:19:34.605028","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4756381,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":12959,"num_write":40,"num_write_kb":4914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"92","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146936","last_change":"2020-10-27 04:19:14.916785","last_active":"2020-10-27 04:19:34.146936","last_peered":"2020-10-27 04:19:34.146936","last_clean":"2020-10-27 04:19:34.146936","last_became_active":"2020-10-27 04:19:14.915920","last_became_peered":"2020-10-27 04:19:14.915920","last_unstale":"2020-10-27 04:19:34.146936","last_undegraded":"2020-10-27 04:19:34.146936","last_fullsized":"2020-10-27 04:19:34.146936","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:52.955136","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:52.955136","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"73'127","reported_seq":"310","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146900","last_change":"2020-10-27 04:19:17.899883","last_active":"2020-10-27 04:19:34.146900","last_peered":"2020-10-27 04:19:34.146900","last_clean":"2020-10-27 04:19:34.146900","last_became_active":"2020-10-27 04:19:17.882175","last_became_peered":"2020-10-27 04:19:17.882175","last_unstale":"2020-10-27 04:19:34.146900","last_undegraded":"2020-10-27 04:19:34.146900","last_fullsized":"2020-10-27 04:19:34.146900","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":127,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9260657,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":31,"num_read_kb":15822,"num_write":191,"num_write_kb":10827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2016840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"88","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608728","last_change":"2020-10-27 04:18:54.277438","last_active":"2020-10-27 04:19:34.608728","last_peered":"2020-10-27 04:19:34.608728","last_clean":"2020-10-27 04:19:34.608728","last_became_active":"2020-10-27 04:18:45.991230","last_became_peered":"2020-10-27 04:18:45.991230","last_unstale":"2020-10-27 04:19:34.608728","last_undegraded":"2020-10-27 04:19:34.608728","last_fullsized":"2020-10-27 04:19:34.608728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.277330","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.277330","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"65'30","reported_seq":"222","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.143009","last_change":"2020-10-27 04:19:27.889205","last_active":"2020-10-27 04:19:34.143009","last_peered":"2020-10-27 04:19:34.143009","last_clean":"2020-10-27 04:19:34.143009","last_became_active":"2020-10-27 04:19:27.887303","last_became_peered":"2020-10-27 04:19:27.887303","last_unstale":"2020-10-27 04:19:34.143009","last_undegraded":"2020-10-27 04:19:34.143009","last_fullsized":"2020-10-27 04:19:34.143009","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5159749,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3999,"num_write":14,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4789995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"106","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:54.406100","last_change":"2020-10-27 04:29:54.406213","last_active":"2020-10-27 04:29:54.406100","last_peered":"2020-10-27 04:29:54.406100","last_clean":"2020-10-27 04:29:54.406100","last_became_active":"2020-10-27 04:18:45.995583","last_became_peered":"2020-10-27 04:18:45.995583","last_unstale":"2020-10-27 04:29:54.406100","last_undegraded":"2020-10-27 04:29:54.406100","last_fullsized":"2020-10-27 04:29:54.406100","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:54.406082","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:54.406082","last_clean_scrub_stamp":"2020-10-27 04:29:54.406082","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"55'50","reported_seq":"228","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147332","last_change":"2020-10-27 04:19:14.929768","last_active":"2020-10-27 04:19:34.147332","last_peered":"2020-10-27 04:19:34.147332","last_clean":"2020-10-27 04:19:34.147332","last_became_active":"2020-10-27 04:19:14.926647","last_became_peered":"2020-10-27 04:19:14.926647","last_unstale":"2020-10-27 04:19:34.147332","last_undegraded":"2020-10-27 04:19:34.147332","last_fullsized":"2020-10-27 04:19:34.147332","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2302559,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":4243,"num_write":22,"num_write_kb":7245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608346","last_change":"2020-10-27 04:18:55.292214","last_active":"2020-10-27 04:19:34.608346","last_peered":"2020-10-27 04:19:34.608346","last_clean":"2020-10-27 04:19:34.608346","last_became_active":"2020-10-27 04:18:45.992519","last_became_peered":"2020-10-27 04:18:45.992519","last_unstale":"2020-10-27 04:19:34.608346","last_undegraded":"2020-10-27 04:19:34.608346","last_fullsized":"2020-10-27 04:19:34.608346","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.292162","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.292162","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'64","reported_seq":"236","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146895","last_change":"2020-10-27 04:19:14.925901","last_active":"2020-10-27 04:19:34.146895","last_peered":"2020-10-27 04:19:34.146895","last_clean":"2020-10-27 04:19:34.146895","last_became_active":"2020-10-27 04:19:14.922144","last_became_peered":"2020-10-27 04:19:14.922144","last_unstale":"2020-10-27 04:19:34.146895","last_undegraded":"2020-10-27 04:19:34.146895","last_fullsized":"2020-10-27 04:19:34.146895","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6100941,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":68,"num_write_kb":6525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"82","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146863","last_change":"2020-10-27 04:19:14.916148","last_active":"2020-10-27 04:19:34.146863","last_peered":"2020-10-27 04:19:34.146863","last_clean":"2020-10-27 04:19:34.146863","last_became_active":"2020-10-27 04:19:14.914681","last_became_peered":"2020-10-27 04:19:14.914681","last_unstale":"2020-10-27 04:19:34.146863","last_undegraded":"2020-10-27 04:19:34.146863","last_fullsized":"2020-10-27 04:19:34.146863","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:17:46.831755","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:17:46.831755","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'110","reported_seq":"284","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146939","last_change":"2020-10-27 04:19:17.889774","last_active":"2020-10-27 04:19:34.146939","last_peered":"2020-10-27 04:19:34.146939","last_clean":"2020-10-27 04:19:34.146939","last_became_active":"2020-10-27 04:19:17.885352","last_became_peered":"2020-10-27 04:19:17.885352","last_unstale":"2020-10-27 04:19:34.146939","last_undegraded":"2020-10-27 04:19:34.146939","last_fullsized":"2020-10-27 04:19:34.146939","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11563216,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1195,"num_write":102,"num_write_kb":11958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"64'43","reported_seq":"227","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.140212","last_change":"2020-10-27 04:18:55.937872","last_active":"2020-10-27 04:19:34.140212","last_peered":"2020-10-27 04:19:34.140212","last_clean":"2020-10-27 04:19:34.140212","last_became_active":"2020-10-27 04:18:14.271076","last_became_peered":"2020-10-27 04:18:14.271076","last_unstale":"2020-10-27 04:19:34.140212","last_undegraded":"2020-10-27 04:19:34.140212","last_fullsized":"2020-10-27 04:19:34.140212","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6050520,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":8972,"num_write":17,"num_write_kb":7783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"104","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:59.341431","last_change":"2020-10-27 04:29:59.341513","last_active":"2020-10-27 04:29:59.341431","last_peered":"2020-10-27 04:29:59.341431","last_clean":"2020-10-27 04:29:59.341431","last_became_active":"2020-10-27 04:18:45.997645","last_became_peered":"2020-10-27 04:18:45.997645","last_unstale":"2020-10-27 04:29:59.341431","last_undegraded":"2020-10-27 04:29:59.341431","last_fullsized":"2020-10-27 04:29:59.341431","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:59.341421","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:59.341421","last_clean_scrub_stamp":"2020-10-27 04:29:59.341421","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'11","reported_seq":"182","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146908","last_change":"2020-10-27 04:19:14.927715","last_active":"2020-10-27 04:19:34.146908","last_peered":"2020-10-27 04:19:34.146908","last_clean":"2020-10-27 04:19:34.146908","last_became_active":"2020-10-27 04:19:14.921996","last_became_peered":"2020-10-27 04:19:14.921996","last_unstale":"2020-10-27 04:19:34.146908","last_undegraded":"2020-10-27 04:19:34.146908","last_fullsized":"2020-10-27 04:19:34.146908","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"89","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:58.601607","last_change":"2020-10-27 04:29:58.601770","last_active":"2020-10-27 04:29:58.601607","last_peered":"2020-10-27 04:29:58.601607","last_clean":"2020-10-27 04:29:58.601607","last_became_active":"2020-10-27 04:19:27.881858","last_became_peered":"2020-10-27 04:19:27.881858","last_unstale":"2020-10-27 04:29:58.601607","last_undegraded":"2020-10-27 04:29:58.601607","last_fullsized":"2020-10-27 04:29:58.601607","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:58.601579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:58.601579","last_clean_scrub_stamp":"2020-10-27 04:29:58.601579","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"73","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142619","last_change":"2020-10-27 04:19:27.888587","last_active":"2020-10-27 04:19:34.142619","last_peered":"2020-10-27 04:19:34.142619","last_clean":"2020-10-27 04:19:34.142619","last_became_active":"2020-10-27 04:19:27.887602","last_became_peered":"2020-10-27 04:19:27.887602","last_unstale":"2020-10-27 04:19:34.142619","last_undegraded":"2020-10-27 04:19:34.142619","last_fullsized":"2020-10-27 04:19:34.142619","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.119312","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.119312","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"63'111","reported_seq":"286","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602263","last_change":"2020-10-27 04:18:55.939797","last_active":"2020-10-27 04:19:34.602263","last_peered":"2020-10-27 04:19:34.602263","last_clean":"2020-10-27 04:19:34.602263","last_became_active":"2020-10-27 04:18:14.266566","last_became_peered":"2020-10-27 04:18:14.266566","last_unstale":"2020-10-27 04:19:34.602263","last_undegraded":"2020-10-27 04:19:34.602263","last_fullsized":"2020-10-27 04:19:34.602263","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14773353,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":2334,"num_write":144,"num_write_kb":16198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608318","last_change":"2020-10-27 04:19:03.292079","last_active":"2020-10-27 04:19:34.608318","last_peered":"2020-10-27 04:19:34.608318","last_clean":"2020-10-27 04:19:34.608318","last_became_active":"2020-10-27 04:18:53.919537","last_became_peered":"2020-10-27 04:18:53.919537","last_unstale":"2020-10-27 04:19:34.608318","last_undegraded":"2020-10-27 04:19:34.608318","last_fullsized":"2020-10-27 04:19:34.608318","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:03.291964","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:03.291964","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"55'28","reported_seq":"175","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142751","last_change":"2020-10-27 04:19:27.888943","last_active":"2020-10-27 04:19:34.142751","last_peered":"2020-10-27 04:19:34.142751","last_clean":"2020-10-27 04:19:34.142751","last_became_active":"2020-10-27 04:19:27.887163","last_became_peered":"2020-10-27 04:19:27.887163","last_unstale":"2020-10-27 04:19:34.142751","last_undegraded":"2020-10-27 04:19:34.142751","last_fullsized":"2020-10-27 04:19:34.142751","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4218557,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1091,"num_write":16,"num_write_kb":4354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608526","last_change":"2020-10-27 04:19:13.189378","last_active":"2020-10-27 04:19:34.608526","last_peered":"2020-10-27 04:19:34.608526","last_clean":"2020-10-27 04:19:34.608526","last_became_active":"2020-10-27 04:18:53.917040","last_became_peered":"2020-10-27 04:18:53.917040","last_unstale":"2020-10-27 04:19:34.608526","last_undegraded":"2020-10-27 04:19:34.608526","last_fullsized":"2020-10-27 04:19:34.608526","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:13.189245","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:13.189245","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"50'22","reported_seq":"195","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427169","last_change":"2020-10-27 04:18:55.910655","last_active":"2020-10-27 04:19:34.427169","last_peered":"2020-10-27 04:19:34.427169","last_clean":"2020-10-27 04:19:34.427169","last_became_active":"2020-10-27 04:18:14.278718","last_became_peered":"2020-10-27 04:18:14.278718","last_unstale":"2020-10-27 04:19:34.427169","last_undegraded":"2020-10-27 04:19:34.427169","last_fullsized":"2020-10-27 04:19:34.427169","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":44,"num_write_kb":1084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"94","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608286","last_change":"2020-10-27 04:19:01.325786","last_active":"2020-10-27 04:19:34.608286","last_peered":"2020-10-27 04:19:34.608286","last_clean":"2020-10-27 04:19:34.608286","last_became_active":"2020-10-27 04:18:53.915676","last_became_peered":"2020-10-27 04:18:53.915676","last_unstale":"2020-10-27 04:19:34.608286","last_undegraded":"2020-10-27 04:19:34.608286","last_fullsized":"2020-10-27 04:19:34.608286","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:01.325624","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:01.325624","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"64'56","reported_seq":"226","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146963","last_change":"2020-10-27 04:19:17.888785","last_active":"2020-10-27 04:19:34.146963","last_peered":"2020-10-27 04:19:34.146963","last_clean":"2020-10-27 04:19:34.146963","last_became_active":"2020-10-27 04:19:17.885684","last_became_peered":"2020-10-27 04:19:17.885684","last_unstale":"2020-10-27 04:19:34.146963","last_undegraded":"2020-10-27 04:19:34.146963","last_fullsized":"2020-10-27 04:19:34.146963","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2026,"num_write":41,"num_write_kb":5997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":"41","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147288","last_change":"2020-10-27 04:19:17.548013","last_active":"2020-10-27 04:19:34.147288","last_peered":"2020-10-27 04:19:34.147288","last_clean":"2020-10-27 04:19:34.147288","last_became_active":"2020-10-27 04:19:14.920337","last_became_peered":"2020-10-27 04:19:14.920337","last_unstale":"2020-10-27 04:19:34.147288","last_undegraded":"2020-10-27 04:19:34.147288","last_fullsized":"2020-10-27 04:19:34.147288","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:17.547918","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:17.547918","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"50'26","reported_seq":"202","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427394","last_change":"2020-10-27 04:18:55.909529","last_active":"2020-10-27 04:19:34.427394","last_peered":"2020-10-27 04:19:34.427394","last_clean":"2020-10-27 04:19:34.427394","last_became_active":"2020-10-27 04:18:14.273127","last_became_peered":"2020-10-27 04:18:14.273127","last_unstale":"2020-10-27 04:19:34.427394","last_undegraded":"2020-10-27 04:19:34.427394","last_fullsized":"2020-10-27 04:19:34.427394","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":547,"num_write":10,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608636","last_change":"2020-10-27 04:19:07.322883","last_active":"2020-10-27 04:19:34.608636","last_peered":"2020-10-27 04:19:34.608636","last_clean":"2020-10-27 04:19:34.608636","last_became_active":"2020-10-27 04:18:53.920956","last_became_peered":"2020-10-27 04:18:53.920956","last_unstale":"2020-10-27 04:19:34.608636","last_undegraded":"2020-10-27 04:19:34.608636","last_fullsized":"2020-10-27 04:19:34.608636","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:07.322732","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:19:07.322732","last_clean_scrub_stamp":"2020-10-27 04:19:07.322732","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"55'62","reported_seq":"242","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.604887","last_change":"2020-10-27 04:18:55.917852","last_active":"2020-10-27 04:19:34.604887","last_peered":"2020-10-27 04:19:34.604887","last_clean":"2020-10-27 04:19:34.604887","last_became_active":"2020-10-27 04:18:14.266855","last_became_peered":"2020-10-27 04:18:14.266855","last_unstale":"2020-10-27 04:19:34.604887","last_undegraded":"2020-10-27 04:19:34.604887","last_fullsized":"2020-10-27 04:19:34.604887","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201717,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":7644,"num_write":22,"num_write_kb":3533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145244160,"data_stored":176377794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575026,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1060788,"kb_used_data":12204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93311052,"statfs":{"total":96636764160,"available":95550517248,"internally_reserved":1073741824,"allocated":12496896,"data_stored":12292194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:15 2020","interfaces":[{"interface":"back","average":{"1min":49.296999999999997,"5min":12.644,"15min":5.9509999999999996},"min":{"1min":1.7649999999999999,"5min":1.2589999999999999,"15min":1.218},"max":{"1min":985.40999999999997,"5min":985.40999999999997,"15min":985.40999999999997},"last":2.218},{"interface":"front","average":{"1min":2.452,"5min":2.4180000000000001,"15min":10.823},"min":{"1min":1.369,"5min":1.2030000000000001,"15min":1.1140000000000001},"max":{"1min":3.79,"5min":3.9300000000000002,"15min":971.95100000000002},"last":2.8319999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":11.887,"15min":8.0559999999999992},"min":{"1min":1.196,"5min":1.196,"15min":1.139},"max":{"1min":3.0030000000000001,"5min":834.52300000000002,"15min":834.52300000000002},"last":1.782},{"interface":"front","average":{"1min":2.6219999999999999,"5min":10.395,"15min":7.609},"min":{"1min":2.1320000000000001,"5min":1.589,"15min":1.095},"max":{"1min":3.218,"5min":621.11199999999997,"15min":675.13099999999997},"last":2.2949999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":37.174999999999997,"5min":9.423,"15min":10.127000000000001},"min":{"1min":1.8260000000000001,"5min":1.5900000000000001,"15min":1.2649999999999999},"max":{"1min":595.22699999999998,"5min":595.22699999999998,"15min":749.68899999999996},"last":2.673},{"interface":"front","average":{"1min":26.971,"5min":7.2380000000000004,"15min":7.1200000000000001},"min":{"1min":1.7769999999999999,"5min":1.5860000000000001,"15min":1.1799999999999999},"max":{"1min":423.61200000000002,"5min":423.61200000000002,"15min":503.75799999999998},"last":1.861}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.7709999999999999,"5min":10.558,"15min":5.4829999999999997},"min":{"1min":2.0529999999999999,"5min":1.9239999999999999,"15min":1.6120000000000001},"max":{"1min":3.29,"5min":810.279,"15min":810.279},"last":2.262},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.6219999999999999,"15min":19.773},"min":{"1min":1.694,"5min":1.6879999999999999,"15min":1.552},"max":{"1min":3.1059999999999999,"5min":3.859,"15min":897.61800000000005},"last":2.5640000000000001}]},{"osd":4,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8399999999999999,"5min":22.492999999999999,"15min":13.928000000000001},"min":{"1min":2.3260000000000001,"5min":1.845,"15min":1.5569999999999999},"max":{"1min":3.2559999999999998,"5min":671.45899999999995,"15min":671.45899999999995},"last":2.887},{"interface":"front","average":{"1min":2.706,"5min":11.789999999999999,"15min":9.1460000000000008},"min":{"1min":1.931,"5min":1.931,"15min":1.665},"max":{"1min":3.2000000000000002,"5min":639.77499999999998,"15min":639.77499999999998},"last":2.758}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":4.444,"15min":5.7809999999999997},"min":{"1min":1.8460000000000001,"5min":1.7,"15min":1.7},"max":{"1min":3.2709999999999999,"5min":159.59399999999999,"15min":735.70500000000004},"last":3.0699999999999998},{"interface":"front","average":{"1min":3.0550000000000002,"5min":16.914999999999999,"15min":16.773},"min":{"1min":2.343,"5min":2.343,"15min":2.1909999999999998},"max":{"1min":3.6560000000000001,"5min":923.20100000000002,"15min":923.20100000000002},"last":2.343}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.3700000000000001,"5min":15.603,"15min":10.483000000000001},"min":{"1min":2.581,"5min":2.581,"15min":2.2679999999999998},"max":{"1min":5.2469999999999999,"5min":992.976,"15min":992.976},"last":679.41899999999998},{"interface":"front","average":{"1min":3.0049999999999999,"5min":6.5810000000000004,"15min":4.5540000000000003},"min":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":1.9530000000000001},"max":{"1min":3.8420000000000001,"5min":390.29300000000001,"15min":390.29300000000001},"last":3.1309999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575025,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1067020,"kb_used_data":18436,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93304820,"statfs":{"total":96636764160,"available":95544135680,"internally_reserved":1073741824,"allocated":18878464,"data_stored":23929015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":2.387,"15min":2.395},"min":{"1min":1.641,"5min":1.571,"15min":1.022},"max":{"1min":3.0219999999999998,"5min":3.4750000000000001,"15min":4.4809999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.5310000000000001,"15min":2.5409999999999999},"min":{"1min":1.53,"5min":1.53,"15min":1.083},"max":{"1min":3.5670000000000002,"5min":3.794,"15min":7.4249999999999998},"last":3.0099999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.109,"15min":2.1930000000000001},"min":{"1min":1.198,"5min":1.175,"15min":1.115},"max":{"1min":2.7810000000000001,"5min":3.2370000000000001,"15min":10.25},"last":2.8330000000000002},{"interface":"front","average":{"1min":2.7709999999999999,"5min":15.755000000000001,"15min":6.71},"min":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.0549999999999999},"max":{"1min":3.649,"5min":1340.4169999999999,"15min":1340.4169999999999},"last":1.724}]},{"osd":2,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.5190000000000001,"5min":2.597,"15min":9.9090000000000007},"min":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.452},"max":{"1min":3.1579999999999999,"5min":3.621,"15min":900.75599999999997},"last":2.367},{"interface":"front","average":{"1min":3.2090000000000001,"5min":2.8330000000000002,"15min":2.7799999999999998},"min":{"1min":2.3420000000000001,"5min":1.605,"15min":1.452},"max":{"1min":4.133,"5min":4.133,"15min":4.7000000000000002},"last":2.7690000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:40 2020","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.8479999999999999,"15min":7.6399999999999997},"min":{"1min":2.0139999999999998,"5min":1.7370000000000001,"15min":1.607},"max":{"1min":3.6840000000000002,"5min":4.452,"15min":637.64599999999996},"last":3.407},{"interface":"front","average":{"1min":3.0979999999999999,"5min":12.949999999999999,"15min":8.9559999999999995},"min":{"1min":2.2290000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"max":{"1min":3.8940000000000001,"5min":1002.468,"15min":1002.468},"last":2.895}]},{"osd":4,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4630000000000001,"5min":2.589,"15min":10.563000000000001},"min":{"1min":1.6459999999999999,"5min":1.462,"15min":1.452},"max":{"1min":3.0089999999999999,"5min":3.7280000000000002,"15min":888.96900000000005},"last":1.79},{"interface":"front","average":{"1min":2.835,"5min":9.891,"15min":7.2279999999999998},"min":{"1min":1.498,"5min":1.498,"15min":1.498},"max":{"1min":3.802,"5min":700.48099999999999,"15min":700.48099999999999},"last":3.1930000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.9039999999999999,"5min":2.9630000000000001,"15min":5.8529999999999998},"min":{"1min":2.2519999999999998,"5min":1.506,"15min":1.506},"max":{"1min":3.7200000000000002,"5min":4.3019999999999996,"15min":552.64099999999996},"last":3.1160000000000001},{"interface":"front","average":{"1min":3.3159999999999998,"5min":12.209,"15min":6.0279999999999996},"min":{"1min":2.3119999999999998,"5min":1.99,"15min":1.488},"max":{"1min":4.0350000000000001,"5min":912.10799999999995,"15min":912.10799999999995},"last":3.355}]},{"osd":7,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":3.0499999999999998,"5min":23.687000000000001,"15min":11.260999999999999},"min":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":1.52},"max":{"1min":3.786,"5min":787.13199999999995,"15min":787.13199999999995},"last":3.2789999999999999},{"interface":"front","average":{"1min":3.569,"5min":3.2679999999999998,"15min":9.1240000000000006},"min":{"1min":2.871,"5min":2.492,"15min":1.966},"max":{"1min":4.6680000000000001,"5min":4.6680000000000001,"15min":1000.52},"last":2.6899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575025,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1068864,"kb_used_data":20280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302976,"statfs":{"total":96636764160,"available":95542247424,"internally_reserved":1073741824,"allocated":20766720,"data_stored":26346047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.6120000000000001,"5min":2.4689999999999999,"15min":6.6299999999999999},"min":{"1min":1.3720000000000001,"5min":1.302,"15min":1.0449999999999999},"max":{"1min":3.3140000000000001,"5min":3.8340000000000001,"15min":787.62400000000002},"last":2.7050000000000001},{"interface":"front","average":{"1min":2.2050000000000001,"5min":12.976000000000001,"15min":14.279999999999999},"min":{"1min":1.4490000000000001,"5min":1.226,"15min":0.89600000000000002},"max":{"1min":3.1429999999999998,"5min":910.65099999999995,"15min":910.65099999999995},"last":2.4209999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":2.5209999999999999,"15min":4.1239999999999997},"min":{"1min":1.591,"5min":1.474,"15min":1.194},"max":{"1min":3.411,"5min":3.8730000000000002,"15min":484.69600000000003},"last":785.25099999999998},{"interface":"front","average":{"1min":2.387,"5min":11.192,"15min":18.960000000000001},"min":{"1min":1.522,"5min":1.522,"15min":1.3979999999999999},"max":{"1min":2.887,"5min":834.95000000000005,"15min":877.15300000000002},"last":3.8279999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":19.675999999999998,"5min":6.1299999999999999,"15min":3.7130000000000001},"min":{"1min":1.972,"5min":1.52,"15min":1.2509999999999999},"max":{"1min":323.01799999999997,"5min":323.01799999999997,"15min":323.01799999999997},"last":2.7610000000000001},{"interface":"front","average":{"1min":2.5329999999999999,"5min":8.093,"15min":6.774},"min":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.238},"max":{"1min":4.3150000000000004,"5min":599.65599999999995,"15min":599.65599999999995},"last":3.9020000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":2.8809999999999998,"15min":2.996},"min":{"1min":1.6950000000000001,"5min":1.661,"15min":1.661},"max":{"1min":3.8069999999999999,"5min":3.899,"15min":4.6299999999999999},"last":2.798},{"interface":"front","average":{"1min":3.0310000000000001,"5min":4.9000000000000004,"15min":4.4480000000000004},"min":{"1min":2.0419999999999998,"5min":1.7190000000000001,"15min":1.3220000000000001},"max":{"1min":4.4039999999999999,"5min":218.10400000000001,"15min":228.80099999999999},"last":3.9889999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.1080000000000001,"5min":3.0630000000000002,"15min":2.968},"min":{"1min":2.399,"5min":2.2349999999999999,"15min":1.5149999999999999},"max":{"1min":3.9729999999999999,"5min":4.1669999999999998,"15min":4.633},"last":3.0600000000000001},{"interface":"front","average":{"1min":2.895,"5min":2.8479999999999999,"15min":5.8499999999999996},"min":{"1min":1.8220000000000001,"5min":1.655,"15min":1.615},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":623.83100000000002},"last":4.0570000000000004}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.2050000000000001,"5min":9.7910000000000004,"15min":10.93},"min":{"1min":2.4510000000000001,"5min":2.2799999999999998,"15min":1.7110000000000001},"max":{"1min":3.895,"5min":727.97400000000005,"15min":816.51300000000003},"last":2.835},{"interface":"front","average":{"1min":2.8940000000000001,"5min":18.178999999999998,"15min":12.436},"min":{"1min":2.2229999999999999,"5min":1.8320000000000001,"15min":1.5700000000000001},"max":{"1min":3.359,"5min":994.68299999999999,"15min":994.68299999999999},"last":2.4940000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.8940000000000001,"5min":6.3639999999999999,"15min":8.2070000000000007},"min":{"1min":2.3940000000000001,"5min":2.0190000000000001,"15min":1.6919999999999999},"max":{"1min":3.496,"5min":346.387,"15min":504.964},"last":2.871},{"interface":"front","average":{"1min":2.988,"5min":3.081,"15min":13.677},"min":{"1min":2.121,"5min":1.9299999999999999,"15min":1.2769999999999999},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":891.01599999999996},"last":4.125}]}]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064972,"kb_used_data":16388,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306868,"statfs":{"total":96636764160,"available":95546232832,"internally_reserved":1073741824,"allocated":16781312,"data_stored":20017524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":14.038,"5min":4.6779999999999999,"15min":3.419},"min":{"1min":1.141,"5min":1.141,"15min":1.054},"max":{"1min":261.02499999999998,"5min":261.02499999999998,"15min":261.02499999999998},"last":1.633},{"interface":"front","average":{"1min":2.7450000000000001,"5min":2.512,"15min":17.265999999999998},"min":{"1min":1.3899999999999999,"5min":1.234,"15min":1.1419999999999999},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":808.98800000000006},"last":2.1509999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":6.6239999999999997,"15min":6.2779999999999996},"min":{"1min":1.702,"5min":1.4510000000000001,"15min":1.0980000000000001},"max":{"1min":4.2809999999999997,"5min":454.637,"15min":654.36699999999996},"last":3.218},{"interface":"front","average":{"1min":3.145,"5min":2.6920000000000002,"15min":5.0830000000000002},"min":{"1min":1.8420000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":949.18700000000001},"last":2.2549999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.351,"5min":7.1449999999999996,"15min":12.432},"min":{"1min":1.4350000000000001,"5min":1.4219999999999999,"15min":1.165},"max":{"1min":5.0810000000000004,"5min":451.41800000000001,"15min":976.43700000000001},"last":2.956},{"interface":"front","average":{"1min":2.6200000000000001,"5min":2.6419999999999999,"15min":11.617000000000001},"min":{"1min":1.754,"5min":1.754,"15min":1.623},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":780.63900000000001},"last":3.3769999999999998}]},{"osd":3,"last update":"Tue Oct 27 04:31:33 2020","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.6269999999999998,"15min":5.4489999999999998},"min":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.3759999999999999},"max":{"1min":3.6629999999999998,"5min":4.8710000000000004,"15min":478.51299999999998},"last":3.5529999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":6.0709999999999997,"15min":3.8820000000000001},"min":{"1min":2.2709999999999999,"5min":1.75,"15min":1.593},"max":{"1min":4.3310000000000004,"5min":344.82600000000002,"15min":344.82600000000002},"last":3.6400000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":13.503,"15min":7.8840000000000003},"min":{"1min":1.9970000000000001,"5min":1.9730000000000001,"15min":1.3819999999999999},"max":{"1min":4.0209999999999999,"5min":800.56899999999996,"15min":800.56899999999996},"last":3.3140000000000001},{"interface":"front","average":{"1min":19.757999999999999,"5min":14.157,"15min":7.9299999999999997},"min":{"1min":2.2829999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"max":{"1min":286.78800000000001,"5min":795.65599999999995,"15min":795.65599999999995},"last":3.5939999999999999}]},{"osd":6,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8820000000000001,"5min":2.7810000000000001,"15min":7.1779999999999999},"min":{"1min":2.0089999999999999,"5min":1.583,"15min":1.583},"max":{"1min":5.2489999999999997,"5min":5.8540000000000001,"15min":872.47000000000003},"last":3.036},{"interface":"front","average":{"1min":3.0329999999999999,"5min":3.0579999999999998,"15min":3.0230000000000001},"min":{"1min":2.089,"5min":2.089,"15min":1.298},"max":{"1min":3.851,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":3.4820000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":6.2850000000000001,"15min":7.04},"min":{"1min":2.194,"5min":1.9630000000000001,"15min":1.9630000000000001},"max":{"1min":5.2960000000000003,"5min":323.93400000000003,"15min":882.08600000000001},"last":3.4300000000000002},{"interface":"front","average":{"1min":3.625,"5min":3.5209999999999999,"15min":6.3170000000000002},"min":{"1min":2.883,"5min":2.2669999999999999,"15min":1.8460000000000001},"max":{"1min":4.3700000000000001,"5min":26.504000000000001,"15min":926.90700000000004},"last":3.6760000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073392,"kb_used_data":24808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93298448,"statfs":{"total":96636764160,"available":95537610752,"internally_reserved":1073741824,"allocated":25403392,"data_stored":30154857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":7.9109999999999996,"15min":4.1260000000000003},"min":{"1min":1.466,"5min":1.1080000000000001,"15min":0.90600000000000003},"max":{"1min":3.4100000000000001,"5min":545.69000000000005,"15min":545.69000000000005},"last":1.8220000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":16.379000000000001,"15min":7.0330000000000004},"min":{"1min":1.6259999999999999,"5min":1.145,"15min":0.80900000000000005},"max":{"1min":3.7789999999999999,"5min":658.68899999999996,"15min":658.68899999999996},"last":2.7639999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":4.7110000000000003,"15min":2.8610000000000002},"min":{"1min":1.3440000000000001,"5min":1.212,"15min":0.76900000000000002},"max":{"1min":3.4079999999999999,"5min":272.61700000000002,"15min":272.61700000000002},"last":2.2610000000000001},{"interface":"front","average":{"1min":2.7120000000000002,"5min":6.8280000000000003,"15min":5.0869999999999997},"min":{"1min":1.893,"5min":1.141,"15min":0.97099999999999997},"max":{"1min":3.718,"5min":251.756,"15min":251.756},"last":2.879}]},{"osd":3,"last update":"Tue Oct 27 04:31:38 2020","interfaces":[{"interface":"back","average":{"1min":60.173000000000002,"5min":13.944000000000001,"15min":13.381},"min":{"1min":1.9079999999999999,"5min":1.365,"15min":1.2030000000000001},"max":{"1min":866.04100000000005,"5min":866.04100000000005,"15min":923.22299999999996},"last":1.7829999999999999},{"interface":"front","average":{"1min":2.7330000000000001,"5min":14.502000000000001,"15min":8.968},"min":{"1min":1.8740000000000001,"5min":1.4970000000000001,"15min":1.26},"max":{"1min":3.5369999999999999,"5min":796.69100000000003,"15min":796.69100000000003},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":42.18,"5min":10.504,"15min":13.028},"min":{"1min":2.0569999999999999,"5min":1.167,"15min":1.167},"max":{"1min":712.15599999999995,"5min":712.15599999999995,"15min":809.928},"last":2.8279999999999998},{"interface":"front","average":{"1min":2.6699999999999999,"5min":2.5510000000000002,"15min":6.2770000000000001},"min":{"1min":1.8460000000000001,"5min":1.401,"15min":1.1379999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":636.61699999999996},"last":2.9830000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":2.593,"15min":6.6749999999999998},"min":{"1min":1.9430000000000001,"5min":1.667,"15min":1.4079999999999999},"max":{"1min":3.7309999999999999,"5min":4.9260000000000002,"15min":577.61300000000006},"last":2.9350000000000001},{"interface":"front","average":{"1min":56.189,"5min":21.494,"15min":9.8610000000000007},"min":{"1min":1.946,"5min":1.5449999999999999,"15min":1.5449999999999999},"max":{"1min":961.779,"5min":961.779,"15min":961.779},"last":3.0800000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.8999999999999999,"5min":9.3569999999999993,"15min":12.731999999999999},"min":{"1min":1.988,"5min":1.8859999999999999,"15min":1.276},"max":{"1min":3.512,"5min":503.51400000000001,"15min":824.16200000000003},"last":3.4289999999999998},{"interface":"front","average":{"1min":3.1749999999999998,"5min":3.1589999999999998,"15min":6.5590000000000002},"min":{"1min":2.4260000000000002,"5min":1.702,"15min":1.5600000000000001},"max":{"1min":3.8519999999999999,"5min":4.6280000000000001,"15min":906.19299999999998},"last":3.871}]},{"osd":7,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":2.8140000000000001,"15min":3.0979999999999999},"min":{"1min":2.222,"5min":1.73,"15min":1.5760000000000001},"max":{"1min":3.77,"5min":4.1669999999999998,"15min":12.324},"last":3.3690000000000002},{"interface":"front","average":{"1min":33.110999999999997,"5min":9.2379999999999995,"15min":6.2720000000000002},"min":{"1min":2.6469999999999998,"5min":1.9750000000000001,"15min":1.4390000000000001},"max":{"1min":510.59899999999999,"5min":510.59899999999999,"15min":510.59899999999999},"last":3.9260000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1065520,"kb_used_data":16936,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306320,"statfs":{"total":96636764160,"available":95545671680,"internally_reserved":1073741824,"allocated":17342464,"data_stored":24243891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.8640000000000001,"15min":2.7629999999999999},"min":{"1min":1.603,"5min":1.226,"15min":0.72199999999999998},"max":{"1min":2.8530000000000002,"5min":2.8530000000000002,"15min":194.143},"last":1.677},{"interface":"front","average":{"1min":2.6930000000000001,"5min":11.317,"15min":10.750999999999999},"min":{"1min":1.9590000000000001,"5min":1.353,"15min":0.81299999999999994},"max":{"1min":4.0609999999999999,"5min":656.79600000000005,"15min":731.65800000000002},"last":2.3330000000000002}]},{"osd":2,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":12.202,"15min":12.997},"min":{"1min":1.5920000000000001,"5min":1.3089999999999999,"15min":0.70199999999999996},"max":{"1min":2.895,"5min":992.125,"15min":992.125},"last":1.7569999999999999},{"interface":"front","average":{"1min":2.7709999999999999,"5min":9.4710000000000001,"15min":5.9850000000000003},"min":{"1min":1.52,"5min":1.3640000000000001,"15min":0.68500000000000005},"max":{"1min":3.6920000000000002,"5min":727.048,"15min":727.048},"last":2.5110000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:36 2020","interfaces":[{"interface":"back","average":{"1min":5.1799999999999997,"5min":3.4060000000000001,"15min":18.274000000000001},"min":{"1min":1.4119999999999999,"5min":1.3720000000000001,"15min":0.92400000000000004},"max":{"1min":61.938000000000002,"5min":63.479999999999997,"15min":879.947},"last":2.9489999999999998},{"interface":"front","average":{"1min":2.6440000000000001,"5min":9.3989999999999991,"15min":8.5289999999999999},"min":{"1min":1.6319999999999999,"5min":1.595,"15min":1.1379999999999999},"max":{"1min":3.4289999999999998,"5min":466.53800000000001,"15min":466.53800000000001},"last":2.2589999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":2.4849999999999999,"15min":3.8580000000000001},"min":{"1min":1.6579999999999999,"5min":1.464,"15min":1.224},"max":{"1min":3.7919999999999998,"5min":3.7919999999999998,"15min":325.25},"last":2.5790000000000002},{"interface":"front","average":{"1min":3.335,"5min":12.461,"15min":6.266},"min":{"1min":2.4980000000000002,"5min":2.121,"15min":1.052},"max":{"1min":4.5529999999999999,"5min":954.96299999999997,"15min":954.96299999999997},"last":3.266}]},{"osd":5,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.5819999999999999,"15min":4.1840000000000002},"min":{"1min":1.675,"5min":1.5349999999999999,"15min":1.173},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":436.06900000000002},"last":3.9769999999999999},{"interface":"front","average":{"1min":3.1739999999999999,"5min":6.9080000000000004,"15min":4.6369999999999996},"min":{"1min":2.0840000000000001,"5min":1.8919999999999999,"15min":1.1990000000000001},"max":{"1min":4.3899999999999997,"5min":425.92899999999997,"15min":425.92899999999997},"last":2.4180000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.96,"5min":2.7599999999999998,"15min":5.3079999999999998},"min":{"1min":2.0219999999999998,"5min":1.855,"15min":1.486},"max":{"1min":3.919,"5min":4.306,"15min":431.327},"last":3.919},{"interface":"front","average":{"1min":3.4300000000000002,"5min":3.2599999999999998,"15min":6.3230000000000004},"min":{"1min":2.5299999999999998,"5min":2.2530000000000001,"15min":1.3879999999999999},"max":{"1min":4.476,"5min":4.476,"15min":800.98800000000006},"last":3.2229999999999999}]},{"osd":7,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":6.6070000000000002,"15min":12.765000000000001},"min":{"1min":2.1960000000000002,"5min":1.976,"15min":1.2789999999999999},"max":{"1min":3.8100000000000001,"5min":448.084,"15min":970.05899999999997},"last":2.4580000000000002},{"interface":"front","average":{"1min":3.1030000000000002,"5min":2.9580000000000002,"15min":5.1280000000000001},"min":{"1min":1.913,"5min":1.913,"15min":1.609},"max":{"1min":4.016,"5min":4.3490000000000002,"15min":498.803},"last":4.016}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064984,"kb_used_data":16400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306856,"statfs":{"total":96636764160,"available":95546220544,"internally_reserved":1073741824,"allocated":16793600,"data_stored":16954654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":53.104999999999997,"5min":13.978999999999999,"15min":11.644},"min":{"1min":1.3340000000000001,"5min":1.1830000000000001,"15min":1.1200000000000001},"max":{"1min":924.39700000000005,"5min":924.39700000000005,"15min":924.39700000000005},"last":3.032},{"interface":"front","average":{"1min":2.355,"5min":2.2330000000000001,"15min":6.7709999999999999},"min":{"1min":1.52,"5min":0.879,"15min":0.68500000000000005},"max":{"1min":2.984,"5min":5.2169999999999996,"15min":948.09299999999996},"last":1.7849999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":14.672000000000001,"15min":6.3040000000000003},"min":{"1min":1.569,"5min":1.123,"15min":0.84699999999999998},"max":{"1min":3.335,"5min":594.51700000000005,"15min":594.51700000000005},"last":1.6739999999999999},{"interface":"front","average":{"1min":2.5619999999999998,"5min":4.0250000000000004,"15min":6.5709999999999997},"min":{"1min":1.7549999999999999,"5min":1.167,"15min":0.95299999999999996},"max":{"1min":3.0720000000000001,"5min":158.51599999999999,"15min":879.13699999999994},"last":2.4180000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:28 2020","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":2.4409999999999998,"15min":5.0830000000000002},"min":{"1min":1.635,"5min":1.2729999999999999,"15min":1.2729999999999999},"max":{"1min":3.4830000000000001,"5min":3.6869999999999998,"15min":615.34500000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":55.475999999999999,"5min":13.064,"15min":6.0110000000000001},"min":{"1min":1.79,"5min":1.1930000000000001,"15min":1.1930000000000001},"max":{"1min":902.85299999999995,"5min":902.85299999999995,"15min":902.85299999999995},"last":2.3679999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.2090000000000001,"5min":2.359,"15min":4.774},"min":{"1min":1.732,"5min":1.54,"15min":1.329},"max":{"1min":3.3359999999999999,"5min":3.766,"15min":551.82500000000005},"last":3.9729999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":19.233000000000001,"15min":14.946},"min":{"1min":1.7669999999999999,"5min":1.7470000000000001,"15min":1.248},"max":{"1min":4.3780000000000001,"5min":953.96799999999996,"15min":953.96799999999996},"last":2.4929999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9790000000000001,"5min":8.7840000000000007,"15min":15.339},"min":{"1min":1.986,"5min":1.7410000000000001,"15min":1.282},"max":{"1min":4.476,"5min":504.077,"15min":975.08900000000006},"last":2.5739999999999998},{"interface":"front","average":{"1min":3.0169999999999999,"5min":12.907999999999999,"15min":11.02},"min":{"1min":2,"5min":1.6439999999999999,"15min":1.518},"max":{"1min":3.9870000000000001,"5min":504.51799999999997,"15min":504.51799999999997},"last":2.774}]},{"osd":6,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":2.7069999999999999,"15min":8.0600000000000005},"min":{"1min":1.821,"5min":1.6890000000000001,"15min":1.506},"max":{"1min":3.427,"5min":3.871,"15min":872.27499999999998},"last":4.04},{"interface":"front","average":{"1min":3.1259999999999999,"5min":3.044,"15min":2.9540000000000002},"min":{"1min":2.2679999999999998,"5min":1.9419999999999999,"15min":1.669},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.5350000000000001}]},{"osd":7,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":10.723000000000001,"15min":6.4370000000000003},"min":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":1.7450000000000001},"max":{"1min":3.891,"5min":510.29599999999999,"15min":510.29599999999999},"last":4.0899999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.1699999999999999,"15min":9.0950000000000006},"min":{"1min":2.706,"5min":2.2109999999999999,"15min":1.504},"max":{"1min":3.831,"5min":5.2839999999999998,"15min":547.11300000000006},"last":2.8210000000000002}]}]},{"osd":3,"up_from":69,"seq":296352743582,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073196,"kb_used_data":24612,"kb_used_omap":7,"kb_used_meta":1048568,"kb_avail":93298644,"statfs":{"total":96636764160,"available":95537811456,"internally_reserved":1073741824,"allocated":25202688,"data_stored":26366732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":1073733794},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":8.8119999999999994,"15min":11.82},"min":{"1min":1.286,"5min":1.014,"15min":1.014},"max":{"1min":2.7869999999999999,"5min":706.96100000000001,"15min":938.43399999999997},"last":2.2669999999999999},{"interface":"front","average":{"1min":44.875999999999998,"5min":19.175000000000001,"15min":7.8029999999999999},"min":{"1min":1.446,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":896.04899999999998,"5min":896.04899999999998,"15min":896.04899999999998},"last":1.915}]},{"osd":1,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":22.134,"5min":6.2039999999999997,"15min":3.9359999999999999},"min":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.069},"max":{"1min":423.48000000000002,"5min":423.48000000000002,"15min":423.48000000000002},"last":2.399},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.3679999999999999,"15min":5.79},"min":{"1min":1.5089999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":3.3540000000000001,"5min":3.472,"15min":678.43899999999996},"last":2.931}]},{"osd":2,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.5699999999999998,"5min":2.4590000000000001,"15min":5.2720000000000002},"min":{"1min":1.6379999999999999,"5min":1.3620000000000001,"15min":1.194},"max":{"1min":3.4199999999999999,"5min":3.7879999999999998,"15min":971.47500000000002},"last":613.66600000000005},{"interface":"front","average":{"1min":5.5800000000000001,"5min":3.29,"15min":2.867},"min":{"1min":2.1019999999999999,"5min":1.25,"15min":1.143},"max":{"1min":63.442,"5min":63.442,"15min":63.442},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":31.300999999999998,"5min":16.239999999999998,"15min":12.332000000000001},"min":{"1min":1.9990000000000001,"5min":1.381,"15min":1.381},"max":{"1min":604.30799999999999,"5min":792.40700000000004,"15min":955.47699999999998},"last":2.468},{"interface":"front","average":{"1min":2.5899999999999999,"5min":4.0339999999999998,"15min":3.0310000000000001},"min":{"1min":1.738,"5min":1.073,"15min":1.073},"max":{"1min":3.556,"5min":151.703,"15min":151.703},"last":2.6699999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":36.460999999999999,"5min":9.4410000000000007,"15min":7.3220000000000001},"min":{"1min":1.8859999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"max":{"1min":710.23599999999999,"5min":710.23599999999999,"15min":710.23599999999999},"last":2.141},{"interface":"front","average":{"1min":15.246,"5min":5.4690000000000003,"15min":7.5579999999999998},"min":{"1min":1.962,"5min":1.554,"15min":1.554},"max":{"1min":261.73599999999999,"5min":261.73599999999999,"15min":607.42600000000004},"last":2.9740000000000002}]},{"osd":6,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":28.733000000000001,"5min":13.307,"15min":6.734},"min":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.55},"max":{"1min":522.255,"5min":522.255,"15min":522.255},"last":2.3610000000000002},{"interface":"front","average":{"1min":2.9729999999999999,"5min":3.0019999999999998,"15min":2.9929999999999999},"min":{"1min":2.2719999999999998,"5min":1.772,"15min":1.772},"max":{"1min":3.8439999999999999,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":3.0339999999999998}]},{"osd":7,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":12.74,"15min":8.8989999999999991},"min":{"1min":2.1600000000000001,"5min":2.0979999999999999,"15min":1.667},"max":{"1min":3.6429999999999998,"5min":607.92200000000003,"15min":867.16300000000001},"last":2.617},{"interface":"front","average":{"1min":3.0920000000000001,"5min":16.859999999999999,"15min":9.5329999999999995},"min":{"1min":2.0529999999999999,"5min":1.5960000000000001,"15min":1.3100000000000001},"max":{"1min":3.8279999999999998,"5min":746.22299999999996,"15min":746.22299999999996},"last":2.5640000000000001}]}]}]}} Ignoring malparsed backtrace: instructing pg 2.ds0 on osd.5 to deep-scrub , 2020-10-27T04:32:32.200 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:32:52.202 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.663 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.771 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:18:52.955136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=52, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:41.487 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-10-27T04:36:41.488 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:36:41.522 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: 2020-10-27T04:36:58.697 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1027/qa/tasks/ceph.py", line 1873, in task Ignoring malparsed backtrace: 2020-10-27T04:37:00.234 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-27T04:37:00.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com , 2020-10-27T04:37:00.235 INFO:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f173b3864f4] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1292) [0x55e87ba78562] Ignoring malparsed backtrace: 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:18.424908", , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:32.937349", , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.040020", , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled" , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done" , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: } , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] Ignoring malparsed backtrace: 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.044446", Ignoring malparsed backtrace: 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg" , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:44.024020", , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:53.982234", Ignoring malparsed backtrace: 2020-10-27T04:18:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:54.002520", , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4294", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.39:0/1113100290", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 673 Ignoring malparsed backtrace: 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:56.931035", , 2020-10-27T04:18:58.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:19:31.121 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 7 1 , reweighted osd.7 to 1 (10000) , 2020-10-27T04:19:32.180 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd primary-affinity 7 1 Ignoring malparsed backtrace: 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus" , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: ], Ignoring malparsed backtrace: 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "mgrmap": { , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 3, , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 4104, Ignoring malparsed backtrace: 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", Ignoring malparsed backtrace: 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", Ignoring malparsed backtrace: 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to persist completed events", , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": {} Ignoring malparsed backtrace: 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", Ignoring malparsed backtrace: 2020-10-27T04:19:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { , 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", , 2020-10-27T04:19:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", Ignoring malparsed backtrace: 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "diskprediction_port": { , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", , 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", Ignoring malparsed backtrace: 2020-10-27T04:19:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", Ignoring malparsed backtrace: 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", Ignoring malparsed backtrace: 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:54.239 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2020-10-27T04:19:54.899 INFO:tasks.ceph:Scrubbing osd.4 Ignoring malparsed backtrace: 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.10 last_scrub_stamp 2020-10-27 04:18:57.342688 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.270 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:22.647 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:22.647 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:21:42.649 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:22:24.804 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:07.878 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:18:54.119312 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:09.996 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.522 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.e , instructing pg 2.es0 on osd.4 to deep-scrub , 2020-10-27T04:24:55.189 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.7 Ignoring malparsed backtrace: 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:26:35.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:26:55.674 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all Ignoring malparsed backtrace: instructing pg 1.0 on osd.1 to deep-scrub , 2020-10-27T04:27:27.955 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.3 , instructing pg 2.3s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:59.786 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.a , instructing pg 1.a on osd.1 to deep-scrub , 2020-10-27T04:30:00.405 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.9 Ignoring malparsed backtrace: 2020-10-27T04:31:08.660 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: dumped all , 2020-10-27T04:32:10.678 INFO:teuthology.orchestra.run.smithi039.stdout: , 2020-10-27T04:32:10.701 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2020-10-27 04:32:09.052368","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8538736,"kb_used_data":150064,"kb_used_omap":7,"kb_used_meta":8388600,"kb_avail":746435984,"statfs":{"total":773094113280,"available":764350447616,"internally_reserved":8589934592,"allocated":153665536,"data_stored":180304914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":8589926562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.11","version":"0'0","reported_seq":"84","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.426833","last_change":"2020-10-27 04:18:55.156939","last_active":"2020-10-27 04:19:34.426833","last_peered":"2020-10-27 04:19:34.426833","last_clean":"2020-10-27 04:19:34.426833","last_became_active":"2020-10-27 04:18:53.912405","last_became_peered":"2020-10-27 04:18:53.912405","last_unstale":"2020-10-27 04:19:34.426833","last_undegraded":"2020-10-27 04:19:34.426833","last_fullsized":"2020-10-27 04:19:34.426833","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.156652","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.156652","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":"38","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602128","last_change":"2020-10-27 04:18:57.342812","last_active":"2020-10-27 04:19:34.602128","last_peered":"2020-10-27 04:19:34.602128","last_clean":"2020-10-27 04:19:34.602128","last_became_active":"2020-10-27 04:18:53.913747","last_became_peered":"2020-10-27 04:18:53.913747","last_unstale":"2020-10-27 04:19:34.602128","last_undegraded":"2020-10-27 04:19:34.602128","last_fullsized":"2020-10-27 04:19:34.602128","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.342688","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.342688","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"39","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427250","last_change":"2020-10-27 04:18:57.141096","last_active":"2020-10-27 04:19:34.427250","last_peered":"2020-10-27 04:19:34.427250","last_clean":"2020-10-27 04:19:34.427250","last_became_active":"2020-10-27 04:18:53.913837","last_became_peered":"2020-10-27 04:18:53.913837","last_unstale":"2020-10-27 04:19:34.427250","last_undegraded":"2020-10-27 04:19:34.427250","last_fullsized":"2020-10-27 04:19:34.427250","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.140975","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.140975","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.f","version":"63'137","reported_seq":"305","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602824","last_change":"2020-10-27 04:18:55.963308","last_active":"2020-10-27 04:19:34.602824","last_peered":"2020-10-27 04:19:34.602824","last_clean":"2020-10-27 04:19:34.602824","last_became_active":"2020-10-27 04:18:14.269110","last_became_peered":"2020-10-27 04:18:14.269110","last_unstale":"2020-10-27 04:19:34.602824","last_undegraded":"2020-10-27 04:19:34.602824","last_fullsized":"2020-10-27 04:19:34.602824","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13731319,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":5580,"num_write":76,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"95","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608614","last_change":"2020-10-27 04:18:56.256991","last_active":"2020-10-27 04:19:34.608614","last_peered":"2020-10-27 04:19:34.608614","last_clean":"2020-10-27 04:19:34.608614","last_became_active":"2020-10-27 04:18:53.917145","last_became_peered":"2020-10-27 04:18:53.917145","last_unstale":"2020-10-27 04:19:34.608614","last_undegraded":"2020-10-27 04:19:34.608614","last_fullsized":"2020-10-27 04:19:34.608614","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:56.256846","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:56.256846","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"55'8","reported_seq":"150","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142999","last_change":"2020-10-27 04:19:27.888673","last_active":"2020-10-27 04:19:34.142999","last_peered":"2020-10-27 04:19:34.142999","last_clean":"2020-10-27 04:19:34.142999","last_became_active":"2020-10-27 04:19:27.887042","last_became_peered":"2020-10-27 04:19:27.887042","last_unstale":"2020-10-27 04:19:34.142999","last_undegraded":"2020-10-27 04:19:34.142999","last_fullsized":"2020-10-27 04:19:34.142999","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":470596,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1382,"num_write":4,"num_write_kb":460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608889","last_change":"2020-10-27 04:18:58.356254","last_active":"2020-10-27 04:19:34.608889","last_peered":"2020-10-27 04:19:34.608889","last_clean":"2020-10-27 04:19:34.608889","last_became_active":"2020-10-27 04:18:45.994526","last_became_peered":"2020-10-27 04:18:45.994526","last_unstale":"2020-10-27 04:19:34.608889","last_undegraded":"2020-10-27 04:19:34.608889","last_fullsized":"2020-10-27 04:19:34.608889","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:58.356124","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:58.356124","last_clean_scrub_stamp":"2020-10-27 04:18:58.356124","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"65'41","reported_seq":"224","reported_epoch":"88","state":"active+clean+scrubbing","last_fresh":"2020-10-27 04:19:34.605028","last_change":"2020-10-27 04:19:14.461624","last_active":"2020-10-27 04:19:34.605028","last_peered":"2020-10-27 04:19:34.605028","last_clean":"2020-10-27 04:19:34.605028","last_became_active":"2020-10-27 04:18:14.276711","last_became_peered":"2020-10-27 04:18:14.276711","last_unstale":"2020-10-27 04:19:34.605028","last_undegraded":"2020-10-27 04:19:34.605028","last_fullsized":"2020-10-27 04:19:34.605028","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4756381,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":12959,"num_write":40,"num_write_kb":4914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"92","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146936","last_change":"2020-10-27 04:19:14.916785","last_active":"2020-10-27 04:19:34.146936","last_peered":"2020-10-27 04:19:34.146936","last_clean":"2020-10-27 04:19:34.146936","last_became_active":"2020-10-27 04:19:14.915920","last_became_peered":"2020-10-27 04:19:14.915920","last_unstale":"2020-10-27 04:19:34.146936","last_undegraded":"2020-10-27 04:19:34.146936","last_fullsized":"2020-10-27 04:19:34.146936","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:52.955136","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:52.955136","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"73'127","reported_seq":"310","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146900","last_change":"2020-10-27 04:19:17.899883","last_active":"2020-10-27 04:19:34.146900","last_peered":"2020-10-27 04:19:34.146900","last_clean":"2020-10-27 04:19:34.146900","last_became_active":"2020-10-27 04:19:17.882175","last_became_peered":"2020-10-27 04:19:17.882175","last_unstale":"2020-10-27 04:19:34.146900","last_undegraded":"2020-10-27 04:19:34.146900","last_fullsized":"2020-10-27 04:19:34.146900","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":127,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9260657,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":31,"num_read_kb":15822,"num_write":191,"num_write_kb":10827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2016840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"88","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608728","last_change":"2020-10-27 04:18:54.277438","last_active":"2020-10-27 04:19:34.608728","last_peered":"2020-10-27 04:19:34.608728","last_clean":"2020-10-27 04:19:34.608728","last_became_active":"2020-10-27 04:18:45.991230","last_became_peered":"2020-10-27 04:18:45.991230","last_unstale":"2020-10-27 04:19:34.608728","last_undegraded":"2020-10-27 04:19:34.608728","last_fullsized":"2020-10-27 04:19:34.608728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.277330","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.277330","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"65'30","reported_seq":"222","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.143009","last_change":"2020-10-27 04:19:27.889205","last_active":"2020-10-27 04:19:34.143009","last_peered":"2020-10-27 04:19:34.143009","last_clean":"2020-10-27 04:19:34.143009","last_became_active":"2020-10-27 04:19:27.887303","last_became_peered":"2020-10-27 04:19:27.887303","last_unstale":"2020-10-27 04:19:34.143009","last_undegraded":"2020-10-27 04:19:34.143009","last_fullsized":"2020-10-27 04:19:34.143009","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5159749,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3999,"num_write":14,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4789995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"106","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:54.406100","last_change":"2020-10-27 04:29:54.406213","last_active":"2020-10-27 04:29:54.406100","last_peered":"2020-10-27 04:29:54.406100","last_clean":"2020-10-27 04:29:54.406100","last_became_active":"2020-10-27 04:18:45.995583","last_became_peered":"2020-10-27 04:18:45.995583","last_unstale":"2020-10-27 04:29:54.406100","last_undegraded":"2020-10-27 04:29:54.406100","last_fullsized":"2020-10-27 04:29:54.406100","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:54.406082","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:54.406082","last_clean_scrub_stamp":"2020-10-27 04:29:54.406082","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"55'50","reported_seq":"228","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147332","last_change":"2020-10-27 04:19:14.929768","last_active":"2020-10-27 04:19:34.147332","last_peered":"2020-10-27 04:19:34.147332","last_clean":"2020-10-27 04:19:34.147332","last_became_active":"2020-10-27 04:19:14.926647","last_became_peered":"2020-10-27 04:19:14.926647","last_unstale":"2020-10-27 04:19:34.147332","last_undegraded":"2020-10-27 04:19:34.147332","last_fullsized":"2020-10-27 04:19:34.147332","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2302559,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":4243,"num_write":22,"num_write_kb":7245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608346","last_change":"2020-10-27 04:18:55.292214","last_active":"2020-10-27 04:19:34.608346","last_peered":"2020-10-27 04:19:34.608346","last_clean":"2020-10-27 04:19:34.608346","last_became_active":"2020-10-27 04:18:45.992519","last_became_peered":"2020-10-27 04:18:45.992519","last_unstale":"2020-10-27 04:19:34.608346","last_undegraded":"2020-10-27 04:19:34.608346","last_fullsized":"2020-10-27 04:19:34.608346","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.292162","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.292162","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'64","reported_seq":"236","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146895","last_change":"2020-10-27 04:19:14.925901","last_active":"2020-10-27 04:19:34.146895","last_peered":"2020-10-27 04:19:34.146895","last_clean":"2020-10-27 04:19:34.146895","last_became_active":"2020-10-27 04:19:14.922144","last_became_peered":"2020-10-27 04:19:14.922144","last_unstale":"2020-10-27 04:19:34.146895","last_undegraded":"2020-10-27 04:19:34.146895","last_fullsized":"2020-10-27 04:19:34.146895","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6100941,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":68,"num_write_kb":6525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"82","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146863","last_change":"2020-10-27 04:19:14.916148","last_active":"2020-10-27 04:19:34.146863","last_peered":"2020-10-27 04:19:34.146863","last_clean":"2020-10-27 04:19:34.146863","last_became_active":"2020-10-27 04:19:14.914681","last_became_peered":"2020-10-27 04:19:14.914681","last_unstale":"2020-10-27 04:19:34.146863","last_undegraded":"2020-10-27 04:19:34.146863","last_fullsized":"2020-10-27 04:19:34.146863","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:17:46.831755","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:17:46.831755","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'110","reported_seq":"284","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146939","last_change":"2020-10-27 04:19:17.889774","last_active":"2020-10-27 04:19:34.146939","last_peered":"2020-10-27 04:19:34.146939","last_clean":"2020-10-27 04:19:34.146939","last_became_active":"2020-10-27 04:19:17.885352","last_became_peered":"2020-10-27 04:19:17.885352","last_unstale":"2020-10-27 04:19:34.146939","last_undegraded":"2020-10-27 04:19:34.146939","last_fullsized":"2020-10-27 04:19:34.146939","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11563216,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1195,"num_write":102,"num_write_kb":11958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"64'43","reported_seq":"227","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.140212","last_change":"2020-10-27 04:18:55.937872","last_active":"2020-10-27 04:19:34.140212","last_peered":"2020-10-27 04:19:34.140212","last_clean":"2020-10-27 04:19:34.140212","last_became_active":"2020-10-27 04:18:14.271076","last_became_peered":"2020-10-27 04:18:14.271076","last_unstale":"2020-10-27 04:19:34.140212","last_undegraded":"2020-10-27 04:19:34.140212","last_fullsized":"2020-10-27 04:19:34.140212","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6050520,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":8972,"num_write":17,"num_write_kb":7783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"104","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:59.341431","last_change":"2020-10-27 04:29:59.341513","last_active":"2020-10-27 04:29:59.341431","last_peered":"2020-10-27 04:29:59.341431","last_clean":"2020-10-27 04:29:59.341431","last_became_active":"2020-10-27 04:18:45.997645","last_became_peered":"2020-10-27 04:18:45.997645","last_unstale":"2020-10-27 04:29:59.341431","last_undegraded":"2020-10-27 04:29:59.341431","last_fullsized":"2020-10-27 04:29:59.341431","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:59.341421","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:59.341421","last_clean_scrub_stamp":"2020-10-27 04:29:59.341421","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'11","reported_seq":"182","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146908","last_change":"2020-10-27 04:19:14.927715","last_active":"2020-10-27 04:19:34.146908","last_peered":"2020-10-27 04:19:34.146908","last_clean":"2020-10-27 04:19:34.146908","last_became_active":"2020-10-27 04:19:14.921996","last_became_peered":"2020-10-27 04:19:14.921996","last_unstale":"2020-10-27 04:19:34.146908","last_undegraded":"2020-10-27 04:19:34.146908","last_fullsized":"2020-10-27 04:19:34.146908","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"89","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:58.601607","last_change":"2020-10-27 04:29:58.601770","last_active":"2020-10-27 04:29:58.601607","last_peered":"2020-10-27 04:29:58.601607","last_clean":"2020-10-27 04:29:58.601607","last_became_active":"2020-10-27 04:19:27.881858","last_became_peered":"2020-10-27 04:19:27.881858","last_unstale":"2020-10-27 04:29:58.601607","last_undegraded":"2020-10-27 04:29:58.601607","last_fullsized":"2020-10-27 04:29:58.601607","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:58.601579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:58.601579","last_clean_scrub_stamp":"2020-10-27 04:29:58.601579","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"73","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142619","last_change":"2020-10-27 04:19:27.888587","last_active":"2020-10-27 04:19:34.142619","last_peered":"2020-10-27 04:19:34.142619","last_clean":"2020-10-27 04:19:34.142619","last_became_active":"2020-10-27 04:19:27.887602","last_became_peered":"2020-10-27 04:19:27.887602","last_unstale":"2020-10-27 04:19:34.142619","last_undegraded":"2020-10-27 04:19:34.142619","last_fullsized":"2020-10-27 04:19:34.142619","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.119312","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.119312","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"63'111","reported_seq":"286","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602263","last_change":"2020-10-27 04:18:55.939797","last_active":"2020-10-27 04:19:34.602263","last_peered":"2020-10-27 04:19:34.602263","last_clean":"2020-10-27 04:19:34.602263","last_became_active":"2020-10-27 04:18:14.266566","last_became_peered":"2020-10-27 04:18:14.266566","last_unstale":"2020-10-27 04:19:34.602263","last_undegraded":"2020-10-27 04:19:34.602263","last_fullsized":"2020-10-27 04:19:34.602263","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14773353,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":2334,"num_write":144,"num_write_kb":16198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608318","last_change":"2020-10-27 04:19:03.292079","last_active":"2020-10-27 04:19:34.608318","last_peered":"2020-10-27 04:19:34.608318","last_clean":"2020-10-27 04:19:34.608318","last_became_active":"2020-10-27 04:18:53.919537","last_became_peered":"2020-10-27 04:18:53.919537","last_unstale":"2020-10-27 04:19:34.608318","last_undegraded":"2020-10-27 04:19:34.608318","last_fullsized":"2020-10-27 04:19:34.608318","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:03.291964","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:03.291964","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"55'28","reported_seq":"175","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142751","last_change":"2020-10-27 04:19:27.888943","last_active":"2020-10-27 04:19:34.142751","last_peered":"2020-10-27 04:19:34.142751","last_clean":"2020-10-27 04:19:34.142751","last_became_active":"2020-10-27 04:19:27.887163","last_became_peered":"2020-10-27 04:19:27.887163","last_unstale":"2020-10-27 04:19:34.142751","last_undegraded":"2020-10-27 04:19:34.142751","last_fullsized":"2020-10-27 04:19:34.142751","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4218557,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1091,"num_write":16,"num_write_kb":4354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608526","last_change":"2020-10-27 04:19:13.189378","last_active":"2020-10-27 04:19:34.608526","last_peered":"2020-10-27 04:19:34.608526","last_clean":"2020-10-27 04:19:34.608526","last_became_active":"2020-10-27 04:18:53.917040","last_became_peered":"2020-10-27 04:18:53.917040","last_unstale":"2020-10-27 04:19:34.608526","last_undegraded":"2020-10-27 04:19:34.608526","last_fullsized":"2020-10-27 04:19:34.608526","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:13.189245","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:13.189245","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"50'22","reported_seq":"195","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427169","last_change":"2020-10-27 04:18:55.910655","last_active":"2020-10-27 04:19:34.427169","last_peered":"2020-10-27 04:19:34.427169","last_clean":"2020-10-27 04:19:34.427169","last_became_active":"2020-10-27 04:18:14.278718","last_became_peered":"2020-10-27 04:18:14.278718","last_unstale":"2020-10-27 04:19:34.427169","last_undegraded":"2020-10-27 04:19:34.427169","last_fullsized":"2020-10-27 04:19:34.427169","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":44,"num_write_kb":1084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"94","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608286","last_change":"2020-10-27 04:19:01.325786","last_active":"2020-10-27 04:19:34.608286","last_peered":"2020-10-27 04:19:34.608286","last_clean":"2020-10-27 04:19:34.608286","last_became_active":"2020-10-27 04:18:53.915676","last_became_peered":"2020-10-27 04:18:53.915676","last_unstale":"2020-10-27 04:19:34.608286","last_undegraded":"2020-10-27 04:19:34.608286","last_fullsized":"2020-10-27 04:19:34.608286","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:01.325624","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:01.325624","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"64'56","reported_seq":"226","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146963","last_change":"2020-10-27 04:19:17.888785","last_active":"2020-10-27 04:19:34.146963","last_peered":"2020-10-27 04:19:34.146963","last_clean":"2020-10-27 04:19:34.146963","last_became_active":"2020-10-27 04:19:17.885684","last_became_peered":"2020-10-27 04:19:17.885684","last_unstale":"2020-10-27 04:19:34.146963","last_undegraded":"2020-10-27 04:19:34.146963","last_fullsized":"2020-10-27 04:19:34.146963","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2026,"num_write":41,"num_write_kb":5997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":"41","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147288","last_change":"2020-10-27 04:19:17.548013","last_active":"2020-10-27 04:19:34.147288","last_peered":"2020-10-27 04:19:34.147288","last_clean":"2020-10-27 04:19:34.147288","last_became_active":"2020-10-27 04:19:14.920337","last_became_peered":"2020-10-27 04:19:14.920337","last_unstale":"2020-10-27 04:19:34.147288","last_undegraded":"2020-10-27 04:19:34.147288","last_fullsized":"2020-10-27 04:19:34.147288","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:17.547918","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:17.547918","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"50'26","reported_seq":"202","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427394","last_change":"2020-10-27 04:18:55.909529","last_active":"2020-10-27 04:19:34.427394","last_peered":"2020-10-27 04:19:34.427394","last_clean":"2020-10-27 04:19:34.427394","last_became_active":"2020-10-27 04:18:14.273127","last_became_peered":"2020-10-27 04:18:14.273127","last_unstale":"2020-10-27 04:19:34.427394","last_undegraded":"2020-10-27 04:19:34.427394","last_fullsized":"2020-10-27 04:19:34.427394","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":547,"num_write":10,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608636","last_change":"2020-10-27 04:19:07.322883","last_active":"2020-10-27 04:19:34.608636","last_peered":"2020-10-27 04:19:34.608636","last_clean":"2020-10-27 04:19:34.608636","last_became_active":"2020-10-27 04:18:53.920956","last_became_peered":"2020-10-27 04:18:53.920956","last_unstale":"2020-10-27 04:19:34.608636","last_undegraded":"2020-10-27 04:19:34.608636","last_fullsized":"2020-10-27 04:19:34.608636","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:07.322732","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:19:07.322732","last_clean_scrub_stamp":"2020-10-27 04:19:07.322732","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"55'62","reported_seq":"242","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.604887","last_change":"2020-10-27 04:18:55.917852","last_active":"2020-10-27 04:19:34.604887","last_peered":"2020-10-27 04:19:34.604887","last_clean":"2020-10-27 04:19:34.604887","last_became_active":"2020-10-27 04:18:14.266855","last_became_peered":"2020-10-27 04:18:14.266855","last_unstale":"2020-10-27 04:19:34.604887","last_undegraded":"2020-10-27 04:19:34.604887","last_fullsized":"2020-10-27 04:19:34.604887","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201717,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":7644,"num_write":22,"num_write_kb":3533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145244160,"data_stored":176377794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575026,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1060788,"kb_used_data":12204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93311052,"statfs":{"total":96636764160,"available":95550517248,"internally_reserved":1073741824,"allocated":12496896,"data_stored":12292194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:15 2020","interfaces":[{"interface":"back","average":{"1min":49.296999999999997,"5min":12.644,"15min":5.9509999999999996},"min":{"1min":1.7649999999999999,"5min":1.2589999999999999,"15min":1.218},"max":{"1min":985.40999999999997,"5min":985.40999999999997,"15min":985.40999999999997},"last":2.218},{"interface":"front","average":{"1min":2.452,"5min":2.4180000000000001,"15min":10.823},"min":{"1min":1.369,"5min":1.2030000000000001,"15min":1.1140000000000001},"max":{"1min":3.79,"5min":3.9300000000000002,"15min":971.95100000000002},"last":2.8319999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":11.887,"15min":8.0559999999999992},"min":{"1min":1.196,"5min":1.196,"15min":1.139},"max":{"1min":3.0030000000000001,"5min":834.52300000000002,"15min":834.52300000000002},"last":1.782},{"interface":"front","average":{"1min":2.6219999999999999,"5min":10.395,"15min":7.609},"min":{"1min":2.1320000000000001,"5min":1.589,"15min":1.095},"max":{"1min":3.218,"5min":621.11199999999997,"15min":675.13099999999997},"last":2.2949999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":37.174999999999997,"5min":9.423,"15min":10.127000000000001},"min":{"1min":1.8260000000000001,"5min":1.5900000000000001,"15min":1.2649999999999999},"max":{"1min":595.22699999999998,"5min":595.22699999999998,"15min":749.68899999999996},"last":2.673},{"interface":"front","average":{"1min":26.971,"5min":7.2380000000000004,"15min":7.1200000000000001},"min":{"1min":1.7769999999999999,"5min":1.5860000000000001,"15min":1.1799999999999999},"max":{"1min":423.61200000000002,"5min":423.61200000000002,"15min":503.75799999999998},"last":1.861}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.7709999999999999,"5min":10.558,"15min":5.4829999999999997},"min":{"1min":2.0529999999999999,"5min":1.9239999999999999,"15min":1.6120000000000001},"max":{"1min":3.29,"5min":810.279,"15min":810.279},"last":2.262},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.6219999999999999,"15min":19.773},"min":{"1min":1.694,"5min":1.6879999999999999,"15min":1.552},"max":{"1min":3.1059999999999999,"5min":3.859,"15min":897.61800000000005},"last":2.5640000000000001}]},{"osd":4,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8399999999999999,"5min":22.492999999999999,"15min":13.928000000000001},"min":{"1min":2.3260000000000001,"5min":1.845,"15min":1.5569999999999999},"max":{"1min":3.2559999999999998,"5min":671.45899999999995,"15min":671.45899999999995},"last":2.887},{"interface":"front","average":{"1min":2.706,"5min":11.789999999999999,"15min":9.1460000000000008},"min":{"1min":1.931,"5min":1.931,"15min":1.665},"max":{"1min":3.2000000000000002,"5min":639.77499999999998,"15min":639.77499999999998},"last":2.758}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":4.444,"15min":5.7809999999999997},"min":{"1min":1.8460000000000001,"5min":1.7,"15min":1.7},"max":{"1min":3.2709999999999999,"5min":159.59399999999999,"15min":735.70500000000004},"last":3.0699999999999998},{"interface":"front","average":{"1min":3.0550000000000002,"5min":16.914999999999999,"15min":16.773},"min":{"1min":2.343,"5min":2.343,"15min":2.1909999999999998},"max":{"1min":3.6560000000000001,"5min":923.20100000000002,"15min":923.20100000000002},"last":2.343}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.3700000000000001,"5min":15.603,"15min":10.483000000000001},"min":{"1min":2.581,"5min":2.581,"15min":2.2679999999999998},"max":{"1min":5.2469999999999999,"5min":992.976,"15min":992.976},"last":679.41899999999998},{"interface":"front","average":{"1min":3.0049999999999999,"5min":6.5810000000000004,"15min":4.5540000000000003},"min":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":1.9530000000000001},"max":{"1min":3.8420000000000001,"5min":390.29300000000001,"15min":390.29300000000001},"last":3.1309999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575025,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1067020,"kb_used_data":18436,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93304820,"statfs":{"total":96636764160,"available":95544135680,"internally_reserved":1073741824,"allocated":18878464,"data_stored":23929015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":2.387,"15min":2.395},"min":{"1min":1.641,"5min":1.571,"15min":1.022},"max":{"1min":3.0219999999999998,"5min":3.4750000000000001,"15min":4.4809999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.5310000000000001,"15min":2.5409999999999999},"min":{"1min":1.53,"5min":1.53,"15min":1.083},"max":{"1min":3.5670000000000002,"5min":3.794,"15min":7.4249999999999998},"last":3.0099999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.109,"15min":2.1930000000000001},"min":{"1min":1.198,"5min":1.175,"15min":1.115},"max":{"1min":2.7810000000000001,"5min":3.2370000000000001,"15min":10.25},"last":2.8330000000000002},{"interface":"front","average":{"1min":2.7709999999999999,"5min":15.755000000000001,"15min":6.71},"min":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.0549999999999999},"max":{"1min":3.649,"5min":1340.4169999999999,"15min":1340.4169999999999},"last":1.724}]},{"osd":2,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.5190000000000001,"5min":2.597,"15min":9.9090000000000007},"min":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.452},"max":{"1min":3.1579999999999999,"5min":3.621,"15min":900.75599999999997},"last":2.367},{"interface":"front","average":{"1min":3.2090000000000001,"5min":2.8330000000000002,"15min":2.7799999999999998},"min":{"1min":2.3420000000000001,"5min":1.605,"15min":1.452},"max":{"1min":4.133,"5min":4.133,"15min":4.7000000000000002},"last":2.7690000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:40 2020","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.8479999999999999,"15min":7.6399999999999997},"min":{"1min":2.0139999999999998,"5min":1.7370000000000001,"15min":1.607},"max":{"1min":3.6840000000000002,"5min":4.452,"15min":637.64599999999996},"last":3.407},{"interface":"front","average":{"1min":3.0979999999999999,"5min":12.949999999999999,"15min":8.9559999999999995},"min":{"1min":2.2290000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"max":{"1min":3.8940000000000001,"5min":1002.468,"15min":1002.468},"last":2.895}]},{"osd":4,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4630000000000001,"5min":2.589,"15min":10.563000000000001},"min":{"1min":1.6459999999999999,"5min":1.462,"15min":1.452},"max":{"1min":3.0089999999999999,"5min":3.7280000000000002,"15min":888.96900000000005},"last":1.79},{"interface":"front","average":{"1min":2.835,"5min":9.891,"15min":7.2279999999999998},"min":{"1min":1.498,"5min":1.498,"15min":1.498},"max":{"1min":3.802,"5min":700.48099999999999,"15min":700.48099999999999},"last":3.1930000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.9039999999999999,"5min":2.9630000000000001,"15min":5.8529999999999998},"min":{"1min":2.2519999999999998,"5min":1.506,"15min":1.506},"max":{"1min":3.7200000000000002,"5min":4.3019999999999996,"15min":552.64099999999996},"last":3.1160000000000001},{"interface":"front","average":{"1min":3.3159999999999998,"5min":12.209,"15min":6.0279999999999996},"min":{"1min":2.3119999999999998,"5min":1.99,"15min":1.488},"max":{"1min":4.0350000000000001,"5min":912.10799999999995,"15min":912.10799999999995},"last":3.355}]},{"osd":7,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":3.0499999999999998,"5min":23.687000000000001,"15min":11.260999999999999},"min":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":1.52},"max":{"1min":3.786,"5min":787.13199999999995,"15min":787.13199999999995},"last":3.2789999999999999},{"interface":"front","average":{"1min":3.569,"5min":3.2679999999999998,"15min":9.1240000000000006},"min":{"1min":2.871,"5min":2.492,"15min":1.966},"max":{"1min":4.6680000000000001,"5min":4.6680000000000001,"15min":1000.52},"last":2.6899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575025,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1068864,"kb_used_data":20280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302976,"statfs":{"total":96636764160,"available":95542247424,"internally_reserved":1073741824,"allocated":20766720,"data_stored":26346047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.6120000000000001,"5min":2.4689999999999999,"15min":6.6299999999999999},"min":{"1min":1.3720000000000001,"5min":1.302,"15min":1.0449999999999999},"max":{"1min":3.3140000000000001,"5min":3.8340000000000001,"15min":787.62400000000002},"last":2.7050000000000001},{"interface":"front","average":{"1min":2.2050000000000001,"5min":12.976000000000001,"15min":14.279999999999999},"min":{"1min":1.4490000000000001,"5min":1.226,"15min":0.89600000000000002},"max":{"1min":3.1429999999999998,"5min":910.65099999999995,"15min":910.65099999999995},"last":2.4209999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":2.5209999999999999,"15min":4.1239999999999997},"min":{"1min":1.591,"5min":1.474,"15min":1.194},"max":{"1min":3.411,"5min":3.8730000000000002,"15min":484.69600000000003},"last":785.25099999999998},{"interface":"front","average":{"1min":2.387,"5min":11.192,"15min":18.960000000000001},"min":{"1min":1.522,"5min":1.522,"15min":1.3979999999999999},"max":{"1min":2.887,"5min":834.95000000000005,"15min":877.15300000000002},"last":3.8279999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":19.675999999999998,"5min":6.1299999999999999,"15min":3.7130000000000001},"min":{"1min":1.972,"5min":1.52,"15min":1.2509999999999999},"max":{"1min":323.01799999999997,"5min":323.01799999999997,"15min":323.01799999999997},"last":2.7610000000000001},{"interface":"front","average":{"1min":2.5329999999999999,"5min":8.093,"15min":6.774},"min":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.238},"max":{"1min":4.3150000000000004,"5min":599.65599999999995,"15min":599.65599999999995},"last":3.9020000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":2.8809999999999998,"15min":2.996},"min":{"1min":1.6950000000000001,"5min":1.661,"15min":1.661},"max":{"1min":3.8069999999999999,"5min":3.899,"15min":4.6299999999999999},"last":2.798},{"interface":"front","average":{"1min":3.0310000000000001,"5min":4.9000000000000004,"15min":4.4480000000000004},"min":{"1min":2.0419999999999998,"5min":1.7190000000000001,"15min":1.3220000000000001},"max":{"1min":4.4039999999999999,"5min":218.10400000000001,"15min":228.80099999999999},"last":3.9889999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.1080000000000001,"5min":3.0630000000000002,"15min":2.968},"min":{"1min":2.399,"5min":2.2349999999999999,"15min":1.5149999999999999},"max":{"1min":3.9729999999999999,"5min":4.1669999999999998,"15min":4.633},"last":3.0600000000000001},{"interface":"front","average":{"1min":2.895,"5min":2.8479999999999999,"15min":5.8499999999999996},"min":{"1min":1.8220000000000001,"5min":1.655,"15min":1.615},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":623.83100000000002},"last":4.0570000000000004}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.2050000000000001,"5min":9.7910000000000004,"15min":10.93},"min":{"1min":2.4510000000000001,"5min":2.2799999999999998,"15min":1.7110000000000001},"max":{"1min":3.895,"5min":727.97400000000005,"15min":816.51300000000003},"last":2.835},{"interface":"front","average":{"1min":2.8940000000000001,"5min":18.178999999999998,"15min":12.436},"min":{"1min":2.2229999999999999,"5min":1.8320000000000001,"15min":1.5700000000000001},"max":{"1min":3.359,"5min":994.68299999999999,"15min":994.68299999999999},"last":2.4940000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.8940000000000001,"5min":6.3639999999999999,"15min":8.2070000000000007},"min":{"1min":2.3940000000000001,"5min":2.0190000000000001,"15min":1.6919999999999999},"max":{"1min":3.496,"5min":346.387,"15min":504.964},"last":2.871},{"interface":"front","average":{"1min":2.988,"5min":3.081,"15min":13.677},"min":{"1min":2.121,"5min":1.9299999999999999,"15min":1.2769999999999999},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":891.01599999999996},"last":4.125}]}]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064972,"kb_used_data":16388,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306868,"statfs":{"total":96636764160,"available":95546232832,"internally_reserved":1073741824,"allocated":16781312,"data_stored":20017524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":14.038,"5min":4.6779999999999999,"15min":3.419},"min":{"1min":1.141,"5min":1.141,"15min":1.054},"max":{"1min":261.02499999999998,"5min":261.02499999999998,"15min":261.02499999999998},"last":1.633},{"interface":"front","average":{"1min":2.7450000000000001,"5min":2.512,"15min":17.265999999999998},"min":{"1min":1.3899999999999999,"5min":1.234,"15min":1.1419999999999999},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":808.98800000000006},"last":2.1509999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":6.6239999999999997,"15min":6.2779999999999996},"min":{"1min":1.702,"5min":1.4510000000000001,"15min":1.0980000000000001},"max":{"1min":4.2809999999999997,"5min":454.637,"15min":654.36699999999996},"last":3.218},{"interface":"front","average":{"1min":3.145,"5min":2.6920000000000002,"15min":5.0830000000000002},"min":{"1min":1.8420000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":949.18700000000001},"last":2.2549999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.351,"5min":7.1449999999999996,"15min":12.432},"min":{"1min":1.4350000000000001,"5min":1.4219999999999999,"15min":1.165},"max":{"1min":5.0810000000000004,"5min":451.41800000000001,"15min":976.43700000000001},"last":2.956},{"interface":"front","average":{"1min":2.6200000000000001,"5min":2.6419999999999999,"15min":11.617000000000001},"min":{"1min":1.754,"5min":1.754,"15min":1.623},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":780.63900000000001},"last":3.3769999999999998}]},{"osd":3,"last update":"Tue Oct 27 04:31:33 2020","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.6269999999999998,"15min":5.4489999999999998},"min":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.3759999999999999},"max":{"1min":3.6629999999999998,"5min":4.8710000000000004,"15min":478.51299999999998},"last":3.5529999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":6.0709999999999997,"15min":3.8820000000000001},"min":{"1min":2.2709999999999999,"5min":1.75,"15min":1.593},"max":{"1min":4.3310000000000004,"5min":344.82600000000002,"15min":344.82600000000002},"last":3.6400000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":13.503,"15min":7.8840000000000003},"min":{"1min":1.9970000000000001,"5min":1.9730000000000001,"15min":1.3819999999999999},"max":{"1min":4.0209999999999999,"5min":800.56899999999996,"15min":800.56899999999996},"last":3.3140000000000001},{"interface":"front","average":{"1min":19.757999999999999,"5min":14.157,"15min":7.9299999999999997},"min":{"1min":2.2829999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"max":{"1min":286.78800000000001,"5min":795.65599999999995,"15min":795.65599999999995},"last":3.5939999999999999}]},{"osd":6,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8820000000000001,"5min":2.7810000000000001,"15min":7.1779999999999999},"min":{"1min":2.0089999999999999,"5min":1.583,"15min":1.583},"max":{"1min":5.2489999999999997,"5min":5.8540000000000001,"15min":872.47000000000003},"last":3.036},{"interface":"front","average":{"1min":3.0329999999999999,"5min":3.0579999999999998,"15min":3.0230000000000001},"min":{"1min":2.089,"5min":2.089,"15min":1.298},"max":{"1min":3.851,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":3.4820000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":6.2850000000000001,"15min":7.04},"min":{"1min":2.194,"5min":1.9630000000000001,"15min":1.9630000000000001},"max":{"1min":5.2960000000000003,"5min":323.93400000000003,"15min":882.08600000000001},"last":3.4300000000000002},{"interface":"front","average":{"1min":3.625,"5min":3.5209999999999999,"15min":6.3170000000000002},"min":{"1min":2.883,"5min":2.2669999999999999,"15min":1.8460000000000001},"max":{"1min":4.3700000000000001,"5min":26.504000000000001,"15min":926.90700000000004},"last":3.6760000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073392,"kb_used_data":24808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93298448,"statfs":{"total":96636764160,"available":95537610752,"internally_reserved":1073741824,"allocated":25403392,"data_stored":30154857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":7.9109999999999996,"15min":4.1260000000000003},"min":{"1min":1.466,"5min":1.1080000000000001,"15min":0.90600000000000003},"max":{"1min":3.4100000000000001,"5min":545.69000000000005,"15min":545.69000000000005},"last":1.8220000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":16.379000000000001,"15min":7.0330000000000004},"min":{"1min":1.6259999999999999,"5min":1.145,"15min":0.80900000000000005},"max":{"1min":3.7789999999999999,"5min":658.68899999999996,"15min":658.68899999999996},"last":2.7639999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":4.7110000000000003,"15min":2.8610000000000002},"min":{"1min":1.3440000000000001,"5min":1.212,"15min":0.76900000000000002},"max":{"1min":3.4079999999999999,"5min":272.61700000000002,"15min":272.61700000000002},"last":2.2610000000000001},{"interface":"front","average":{"1min":2.7120000000000002,"5min":6.8280000000000003,"15min":5.0869999999999997},"min":{"1min":1.893,"5min":1.141,"15min":0.97099999999999997},"max":{"1min":3.718,"5min":251.756,"15min":251.756},"last":2.879}]},{"osd":3,"last update":"Tue Oct 27 04:31:38 2020","interfaces":[{"interface":"back","average":{"1min":60.173000000000002,"5min":13.944000000000001,"15min":13.381},"min":{"1min":1.9079999999999999,"5min":1.365,"15min":1.2030000000000001},"max":{"1min":866.04100000000005,"5min":866.04100000000005,"15min":923.22299999999996},"last":1.7829999999999999},{"interface":"front","average":{"1min":2.7330000000000001,"5min":14.502000000000001,"15min":8.968},"min":{"1min":1.8740000000000001,"5min":1.4970000000000001,"15min":1.26},"max":{"1min":3.5369999999999999,"5min":796.69100000000003,"15min":796.69100000000003},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":42.18,"5min":10.504,"15min":13.028},"min":{"1min":2.0569999999999999,"5min":1.167,"15min":1.167},"max":{"1min":712.15599999999995,"5min":712.15599999999995,"15min":809.928},"last":2.8279999999999998},{"interface":"front","average":{"1min":2.6699999999999999,"5min":2.5510000000000002,"15min":6.2770000000000001},"min":{"1min":1.8460000000000001,"5min":1.401,"15min":1.1379999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":636.61699999999996},"last":2.9830000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":2.593,"15min":6.6749999999999998},"min":{"1min":1.9430000000000001,"5min":1.667,"15min":1.4079999999999999},"max":{"1min":3.7309999999999999,"5min":4.9260000000000002,"15min":577.61300000000006},"last":2.9350000000000001},{"interface":"front","average":{"1min":56.189,"5min":21.494,"15min":9.8610000000000007},"min":{"1min":1.946,"5min":1.5449999999999999,"15min":1.5449999999999999},"max":{"1min":961.779,"5min":961.779,"15min":961.779},"last":3.0800000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.8999999999999999,"5min":9.3569999999999993,"15min":12.731999999999999},"min":{"1min":1.988,"5min":1.8859999999999999,"15min":1.276},"max":{"1min":3.512,"5min":503.51400000000001,"15min":824.16200000000003},"last":3.4289999999999998},{"interface":"front","average":{"1min":3.1749999999999998,"5min":3.1589999999999998,"15min":6.5590000000000002},"min":{"1min":2.4260000000000002,"5min":1.702,"15min":1.5600000000000001},"max":{"1min":3.8519999999999999,"5min":4.6280000000000001,"15min":906.19299999999998},"last":3.871}]},{"osd":7,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":2.8140000000000001,"15min":3.0979999999999999},"min":{"1min":2.222,"5min":1.73,"15min":1.5760000000000001},"max":{"1min":3.77,"5min":4.1669999999999998,"15min":12.324},"last":3.3690000000000002},{"interface":"front","average":{"1min":33.110999999999997,"5min":9.2379999999999995,"15min":6.2720000000000002},"min":{"1min":2.6469999999999998,"5min":1.9750000000000001,"15min":1.4390000000000001},"max":{"1min":510.59899999999999,"5min":510.59899999999999,"15min":510.59899999999999},"last":3.9260000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1065520,"kb_used_data":16936,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306320,"statfs":{"total":96636764160,"available":95545671680,"internally_reserved":1073741824,"allocated":17342464,"data_stored":24243891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.8640000000000001,"15min":2.7629999999999999},"min":{"1min":1.603,"5min":1.226,"15min":0.72199999999999998},"max":{"1min":2.8530000000000002,"5min":2.8530000000000002,"15min":194.143},"last":1.677},{"interface":"front","average":{"1min":2.6930000000000001,"5min":11.317,"15min":10.750999999999999},"min":{"1min":1.9590000000000001,"5min":1.353,"15min":0.81299999999999994},"max":{"1min":4.0609999999999999,"5min":656.79600000000005,"15min":731.65800000000002},"last":2.3330000000000002}]},{"osd":2,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":12.202,"15min":12.997},"min":{"1min":1.5920000000000001,"5min":1.3089999999999999,"15min":0.70199999999999996},"max":{"1min":2.895,"5min":992.125,"15min":992.125},"last":1.7569999999999999},{"interface":"front","average":{"1min":2.7709999999999999,"5min":9.4710000000000001,"15min":5.9850000000000003},"min":{"1min":1.52,"5min":1.3640000000000001,"15min":0.68500000000000005},"max":{"1min":3.6920000000000002,"5min":727.048,"15min":727.048},"last":2.5110000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:36 2020","interfaces":[{"interface":"back","average":{"1min":5.1799999999999997,"5min":3.4060000000000001,"15min":18.274000000000001},"min":{"1min":1.4119999999999999,"5min":1.3720000000000001,"15min":0.92400000000000004},"max":{"1min":61.938000000000002,"5min":63.479999999999997,"15min":879.947},"last":2.9489999999999998},{"interface":"front","average":{"1min":2.6440000000000001,"5min":9.3989999999999991,"15min":8.5289999999999999},"min":{"1min":1.6319999999999999,"5min":1.595,"15min":1.1379999999999999},"max":{"1min":3.4289999999999998,"5min":466.53800000000001,"15min":466.53800000000001},"last":2.2589999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":2.4849999999999999,"15min":3.8580000000000001},"min":{"1min":1.6579999999999999,"5min":1.464,"15min":1.224},"max":{"1min":3.7919999999999998,"5min":3.7919999999999998,"15min":325.25},"last":2.5790000000000002},{"interface":"front","average":{"1min":3.335,"5min":12.461,"15min":6.266},"min":{"1min":2.4980000000000002,"5min":2.121,"15min":1.052},"max":{"1min":4.5529999999999999,"5min":954.96299999999997,"15min":954.96299999999997},"last":3.266}]},{"osd":5,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.5819999999999999,"15min":4.1840000000000002},"min":{"1min":1.675,"5min":1.5349999999999999,"15min":1.173},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":436.06900000000002},"last":3.9769999999999999},{"interface":"front","average":{"1min":3.1739999999999999,"5min":6.9080000000000004,"15min":4.6369999999999996},"min":{"1min":2.0840000000000001,"5min":1.8919999999999999,"15min":1.1990000000000001},"max":{"1min":4.3899999999999997,"5min":425.92899999999997,"15min":425.92899999999997},"last":2.4180000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.96,"5min":2.7599999999999998,"15min":5.3079999999999998},"min":{"1min":2.0219999999999998,"5min":1.855,"15min":1.486},"max":{"1min":3.919,"5min":4.306,"15min":431.327},"last":3.919},{"interface":"front","average":{"1min":3.4300000000000002,"5min":3.2599999999999998,"15min":6.3230000000000004},"min":{"1min":2.5299999999999998,"5min":2.2530000000000001,"15min":1.3879999999999999},"max":{"1min":4.476,"5min":4.476,"15min":800.98800000000006},"last":3.2229999999999999}]},{"osd":7,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":6.6070000000000002,"15min":12.765000000000001},"min":{"1min":2.1960000000000002,"5min":1.976,"15min":1.2789999999999999},"max":{"1min":3.8100000000000001,"5min":448.084,"15min":970.05899999999997},"last":2.4580000000000002},{"interface":"front","average":{"1min":3.1030000000000002,"5min":2.9580000000000002,"15min":5.1280000000000001},"min":{"1min":1.913,"5min":1.913,"15min":1.609},"max":{"1min":4.016,"5min":4.3490000000000002,"15min":498.803},"last":4.016}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064984,"kb_used_data":16400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306856,"statfs":{"total":96636764160,"available":95546220544,"internally_reserved":1073741824,"allocated":16793600,"data_stored":16954654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":53.104999999999997,"5min":13.978999999999999,"15min":11.644},"min":{"1min":1.3340000000000001,"5min":1.1830000000000001,"15min":1.1200000000000001},"max":{"1min":924.39700000000005,"5min":924.39700000000005,"15min":924.39700000000005},"last":3.032},{"interface":"front","average":{"1min":2.355,"5min":2.2330000000000001,"15min":6.7709999999999999},"min":{"1min":1.52,"5min":0.879,"15min":0.68500000000000005},"max":{"1min":2.984,"5min":5.2169999999999996,"15min":948.09299999999996},"last":1.7849999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":14.672000000000001,"15min":6.3040000000000003},"min":{"1min":1.569,"5min":1.123,"15min":0.84699999999999998},"max":{"1min":3.335,"5min":594.51700000000005,"15min":594.51700000000005},"last":1.6739999999999999},{"interface":"front","average":{"1min":2.5619999999999998,"5min":4.0250000000000004,"15min":6.5709999999999997},"min":{"1min":1.7549999999999999,"5min":1.167,"15min":0.95299999999999996},"max":{"1min":3.0720000000000001,"5min":158.51599999999999,"15min":879.13699999999994},"last":2.4180000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:28 2020","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":2.4409999999999998,"15min":5.0830000000000002},"min":{"1min":1.635,"5min":1.2729999999999999,"15min":1.2729999999999999},"max":{"1min":3.4830000000000001,"5min":3.6869999999999998,"15min":615.34500000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":55.475999999999999,"5min":13.064,"15min":6.0110000000000001},"min":{"1min":1.79,"5min":1.1930000000000001,"15min":1.1930000000000001},"max":{"1min":902.85299999999995,"5min":902.85299999999995,"15min":902.85299999999995},"last":2.3679999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.2090000000000001,"5min":2.359,"15min":4.774},"min":{"1min":1.732,"5min":1.54,"15min":1.329},"max":{"1min":3.3359999999999999,"5min":3.766,"15min":551.82500000000005},"last":3.9729999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":19.233000000000001,"15min":14.946},"min":{"1min":1.7669999999999999,"5min":1.7470000000000001,"15min":1.248},"max":{"1min":4.3780000000000001,"5min":953.96799999999996,"15min":953.96799999999996},"last":2.4929999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9790000000000001,"5min":8.7840000000000007,"15min":15.339},"min":{"1min":1.986,"5min":1.7410000000000001,"15min":1.282},"max":{"1min":4.476,"5min":504.077,"15min":975.08900000000006},"last":2.5739999999999998},{"interface":"front","average":{"1min":3.0169999999999999,"5min":12.907999999999999,"15min":11.02},"min":{"1min":2,"5min":1.6439999999999999,"15min":1.518},"max":{"1min":3.9870000000000001,"5min":504.51799999999997,"15min":504.51799999999997},"last":2.774}]},{"osd":6,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":2.7069999999999999,"15min":8.0600000000000005},"min":{"1min":1.821,"5min":1.6890000000000001,"15min":1.506},"max":{"1min":3.427,"5min":3.871,"15min":872.27499999999998},"last":4.04},{"interface":"front","average":{"1min":3.1259999999999999,"5min":3.044,"15min":2.9540000000000002},"min":{"1min":2.2679999999999998,"5min":1.9419999999999999,"15min":1.669},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.5350000000000001}]},{"osd":7,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":10.723000000000001,"15min":6.4370000000000003},"min":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":1.7450000000000001},"max":{"1min":3.891,"5min":510.29599999999999,"15min":510.29599999999999},"last":4.0899999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.1699999999999999,"15min":9.0950000000000006},"min":{"1min":2.706,"5min":2.2109999999999999,"15min":1.504},"max":{"1min":3.831,"5min":5.2839999999999998,"15min":547.11300000000006},"last":2.8210000000000002}]}]},{"osd":3,"up_from":69,"seq":296352743582,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073196,"kb_used_data":24612,"kb_used_omap":7,"kb_used_meta":1048568,"kb_avail":93298644,"statfs":{"total":96636764160,"available":95537811456,"internally_reserved":1073741824,"allocated":25202688,"data_stored":26366732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":1073733794},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":8.8119999999999994,"15min":11.82},"min":{"1min":1.286,"5min":1.014,"15min":1.014},"max":{"1min":2.7869999999999999,"5min":706.96100000000001,"15min":938.43399999999997},"last":2.2669999999999999},{"interface":"front","average":{"1min":44.875999999999998,"5min":19.175000000000001,"15min":7.8029999999999999},"min":{"1min":1.446,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":896.04899999999998,"5min":896.04899999999998,"15min":896.04899999999998},"last":1.915}]},{"osd":1,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":22.134,"5min":6.2039999999999997,"15min":3.9359999999999999},"min":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.069},"max":{"1min":423.48000000000002,"5min":423.48000000000002,"15min":423.48000000000002},"last":2.399},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.3679999999999999,"15min":5.79},"min":{"1min":1.5089999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":3.3540000000000001,"5min":3.472,"15min":678.43899999999996},"last":2.931}]},{"osd":2,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.5699999999999998,"5min":2.4590000000000001,"15min":5.2720000000000002},"min":{"1min":1.6379999999999999,"5min":1.3620000000000001,"15min":1.194},"max":{"1min":3.4199999999999999,"5min":3.7879999999999998,"15min":971.47500000000002},"last":613.66600000000005},{"interface":"front","average":{"1min":5.5800000000000001,"5min":3.29,"15min":2.867},"min":{"1min":2.1019999999999999,"5min":1.25,"15min":1.143},"max":{"1min":63.442,"5min":63.442,"15min":63.442},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":31.300999999999998,"5min":16.239999999999998,"15min":12.332000000000001},"min":{"1min":1.9990000000000001,"5min":1.381,"15min":1.381},"max":{"1min":604.30799999999999,"5min":792.40700000000004,"15min":955.47699999999998},"last":2.468},{"interface":"front","average":{"1min":2.5899999999999999,"5min":4.0339999999999998,"15min":3.0310000000000001},"min":{"1min":1.738,"5min":1.073,"15min":1.073},"max":{"1min":3.556,"5min":151.703,"15min":151.703},"last":2.6699999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":36.460999999999999,"5min":9.4410000000000007,"15min":7.3220000000000001},"min":{"1min":1.8859999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"max":{"1min":710.23599999999999,"5min":710.23599999999999,"15min":710.23599999999999},"last":2.141},{"interface":"front","average":{"1min":15.246,"5min":5.4690000000000003,"15min":7.5579999999999998},"min":{"1min":1.962,"5min":1.554,"15min":1.554},"max":{"1min":261.73599999999999,"5min":261.73599999999999,"15min":607.42600000000004},"last":2.9740000000000002}]},{"osd":6,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":28.733000000000001,"5min":13.307,"15min":6.734},"min":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.55},"max":{"1min":522.255,"5min":522.255,"15min":522.255},"last":2.3610000000000002},{"interface":"front","average":{"1min":2.9729999999999999,"5min":3.0019999999999998,"15min":2.9929999999999999},"min":{"1min":2.2719999999999998,"5min":1.772,"15min":1.772},"max":{"1min":3.8439999999999999,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":3.0339999999999998}]},{"osd":7,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":12.74,"15min":8.8989999999999991},"min":{"1min":2.1600000000000001,"5min":2.0979999999999999,"15min":1.667},"max":{"1min":3.6429999999999998,"5min":607.92200000000003,"15min":867.16300000000001},"last":2.617},{"interface":"front","average":{"1min":3.0920000000000001,"5min":16.859999999999999,"15min":9.5329999999999995},"min":{"1min":2.0529999999999999,"5min":1.5960000000000001,"15min":1.3100000000000001},"max":{"1min":3.8279999999999998,"5min":746.22299999999996,"15min":746.22299999999996},"last":2.5640000000000001}]}]}]}} Ignoring malparsed backtrace: instructing pg 2.ds0 on osd.5 to deep-scrub , 2020-10-27T04:32:32.200 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:32:52.202 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.663 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.771 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:18:52.955136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=52, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:41.487 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-10-27T04:36:41.488 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:36:41.522 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: 2020-10-27T04:36:58.697 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1027/qa/tasks/ceph.py", line 1873, in task Ignoring malparsed backtrace: 2020-10-27T04:37:00.234 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-27T04:37:00.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com , 2020-10-27T04:37:00.235 INFO:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f173b3864f4] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1292) [0x55e87ba78562] Ignoring malparsed backtrace: 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:18.424908", , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:32.937349", , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.040020", , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled" , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done" , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: } , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] Ignoring malparsed backtrace: 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.044446", Ignoring malparsed backtrace: 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg" , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:44.024020", , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:53.982234", Ignoring malparsed backtrace: 2020-10-27T04:18:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:54.002520", , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4294", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.39:0/1113100290", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 673 Ignoring malparsed backtrace: 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:56.931035", , 2020-10-27T04:18:58.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:19:31.121 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 7 1 , reweighted osd.7 to 1 (10000) , 2020-10-27T04:19:32.180 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd primary-affinity 7 1 Ignoring malparsed backtrace: 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus" , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: ], Ignoring malparsed backtrace: 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "mgrmap": { , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 3, , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 4104, Ignoring malparsed backtrace: 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", Ignoring malparsed backtrace: 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", Ignoring malparsed backtrace: 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to persist completed events", , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": {} Ignoring malparsed backtrace: 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", Ignoring malparsed backtrace: 2020-10-27T04:19:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { , 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", , 2020-10-27T04:19:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", Ignoring malparsed backtrace: 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "diskprediction_port": { , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", , 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", Ignoring malparsed backtrace: 2020-10-27T04:19:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", Ignoring malparsed backtrace: 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", Ignoring malparsed backtrace: 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:54.239 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2020-10-27T04:19:54.899 INFO:tasks.ceph:Scrubbing osd.4 Ignoring malparsed backtrace: 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.10 last_scrub_stamp 2020-10-27 04:18:57.342688 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.270 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:22.647 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:22.647 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:21:42.649 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:22:24.804 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:07.878 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:18:54.119312 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:09.996 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.522 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.e , instructing pg 2.es0 on osd.4 to deep-scrub , 2020-10-27T04:24:55.189 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.7 Ignoring malparsed backtrace: 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:26:35.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:26:55.674 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all Ignoring malparsed backtrace: instructing pg 1.0 on osd.1 to deep-scrub , 2020-10-27T04:27:27.955 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.3 , instructing pg 2.3s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:59.786 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.a , instructing pg 1.a on osd.1 to deep-scrub , 2020-10-27T04:30:00.405 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.9 Ignoring malparsed backtrace: 2020-10-27T04:31:08.660 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: dumped all , 2020-10-27T04:32:10.678 INFO:teuthology.orchestra.run.smithi039.stdout: , 2020-10-27T04:32:10.701 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2020-10-27 04:32:09.052368","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8538736,"kb_used_data":150064,"kb_used_omap":7,"kb_used_meta":8388600,"kb_avail":746435984,"statfs":{"total":773094113280,"available":764350447616,"internally_reserved":8589934592,"allocated":153665536,"data_stored":180304914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":8589926562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.11","version":"0'0","reported_seq":"84","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.426833","last_change":"2020-10-27 04:18:55.156939","last_active":"2020-10-27 04:19:34.426833","last_peered":"2020-10-27 04:19:34.426833","last_clean":"2020-10-27 04:19:34.426833","last_became_active":"2020-10-27 04:18:53.912405","last_became_peered":"2020-10-27 04:18:53.912405","last_unstale":"2020-10-27 04:19:34.426833","last_undegraded":"2020-10-27 04:19:34.426833","last_fullsized":"2020-10-27 04:19:34.426833","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.156652","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.156652","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":"38","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602128","last_change":"2020-10-27 04:18:57.342812","last_active":"2020-10-27 04:19:34.602128","last_peered":"2020-10-27 04:19:34.602128","last_clean":"2020-10-27 04:19:34.602128","last_became_active":"2020-10-27 04:18:53.913747","last_became_peered":"2020-10-27 04:18:53.913747","last_unstale":"2020-10-27 04:19:34.602128","last_undegraded":"2020-10-27 04:19:34.602128","last_fullsized":"2020-10-27 04:19:34.602128","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.342688","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.342688","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"39","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427250","last_change":"2020-10-27 04:18:57.141096","last_active":"2020-10-27 04:19:34.427250","last_peered":"2020-10-27 04:19:34.427250","last_clean":"2020-10-27 04:19:34.427250","last_became_active":"2020-10-27 04:18:53.913837","last_became_peered":"2020-10-27 04:18:53.913837","last_unstale":"2020-10-27 04:19:34.427250","last_undegraded":"2020-10-27 04:19:34.427250","last_fullsized":"2020-10-27 04:19:34.427250","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.140975","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.140975","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.f","version":"63'137","reported_seq":"305","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602824","last_change":"2020-10-27 04:18:55.963308","last_active":"2020-10-27 04:19:34.602824","last_peered":"2020-10-27 04:19:34.602824","last_clean":"2020-10-27 04:19:34.602824","last_became_active":"2020-10-27 04:18:14.269110","last_became_peered":"2020-10-27 04:18:14.269110","last_unstale":"2020-10-27 04:19:34.602824","last_undegraded":"2020-10-27 04:19:34.602824","last_fullsized":"2020-10-27 04:19:34.602824","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13731319,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":5580,"num_write":76,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"95","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608614","last_change":"2020-10-27 04:18:56.256991","last_active":"2020-10-27 04:19:34.608614","last_peered":"2020-10-27 04:19:34.608614","last_clean":"2020-10-27 04:19:34.608614","last_became_active":"2020-10-27 04:18:53.917145","last_became_peered":"2020-10-27 04:18:53.917145","last_unstale":"2020-10-27 04:19:34.608614","last_undegraded":"2020-10-27 04:19:34.608614","last_fullsized":"2020-10-27 04:19:34.608614","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:56.256846","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:56.256846","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"55'8","reported_seq":"150","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142999","last_change":"2020-10-27 04:19:27.888673","last_active":"2020-10-27 04:19:34.142999","last_peered":"2020-10-27 04:19:34.142999","last_clean":"2020-10-27 04:19:34.142999","last_became_active":"2020-10-27 04:19:27.887042","last_became_peered":"2020-10-27 04:19:27.887042","last_unstale":"2020-10-27 04:19:34.142999","last_undegraded":"2020-10-27 04:19:34.142999","last_fullsized":"2020-10-27 04:19:34.142999","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":470596,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1382,"num_write":4,"num_write_kb":460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608889","last_change":"2020-10-27 04:18:58.356254","last_active":"2020-10-27 04:19:34.608889","last_peered":"2020-10-27 04:19:34.608889","last_clean":"2020-10-27 04:19:34.608889","last_became_active":"2020-10-27 04:18:45.994526","last_became_peered":"2020-10-27 04:18:45.994526","last_unstale":"2020-10-27 04:19:34.608889","last_undegraded":"2020-10-27 04:19:34.608889","last_fullsized":"2020-10-27 04:19:34.608889","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:58.356124","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:58.356124","last_clean_scrub_stamp":"2020-10-27 04:18:58.356124","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"65'41","reported_seq":"224","reported_epoch":"88","state":"active+clean+scrubbing","last_fresh":"2020-10-27 04:19:34.605028","last_change":"2020-10-27 04:19:14.461624","last_active":"2020-10-27 04:19:34.605028","last_peered":"2020-10-27 04:19:34.605028","last_clean":"2020-10-27 04:19:34.605028","last_became_active":"2020-10-27 04:18:14.276711","last_became_peered":"2020-10-27 04:18:14.276711","last_unstale":"2020-10-27 04:19:34.605028","last_undegraded":"2020-10-27 04:19:34.605028","last_fullsized":"2020-10-27 04:19:34.605028","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4756381,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":12959,"num_write":40,"num_write_kb":4914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"92","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146936","last_change":"2020-10-27 04:19:14.916785","last_active":"2020-10-27 04:19:34.146936","last_peered":"2020-10-27 04:19:34.146936","last_clean":"2020-10-27 04:19:34.146936","last_became_active":"2020-10-27 04:19:14.915920","last_became_peered":"2020-10-27 04:19:14.915920","last_unstale":"2020-10-27 04:19:34.146936","last_undegraded":"2020-10-27 04:19:34.146936","last_fullsized":"2020-10-27 04:19:34.146936","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:52.955136","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:52.955136","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"73'127","reported_seq":"310","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146900","last_change":"2020-10-27 04:19:17.899883","last_active":"2020-10-27 04:19:34.146900","last_peered":"2020-10-27 04:19:34.146900","last_clean":"2020-10-27 04:19:34.146900","last_became_active":"2020-10-27 04:19:17.882175","last_became_peered":"2020-10-27 04:19:17.882175","last_unstale":"2020-10-27 04:19:34.146900","last_undegraded":"2020-10-27 04:19:34.146900","last_fullsized":"2020-10-27 04:19:34.146900","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":127,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9260657,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":31,"num_read_kb":15822,"num_write":191,"num_write_kb":10827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2016840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"88","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608728","last_change":"2020-10-27 04:18:54.277438","last_active":"2020-10-27 04:19:34.608728","last_peered":"2020-10-27 04:19:34.608728","last_clean":"2020-10-27 04:19:34.608728","last_became_active":"2020-10-27 04:18:45.991230","last_became_peered":"2020-10-27 04:18:45.991230","last_unstale":"2020-10-27 04:19:34.608728","last_undegraded":"2020-10-27 04:19:34.608728","last_fullsized":"2020-10-27 04:19:34.608728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.277330","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.277330","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"65'30","reported_seq":"222","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.143009","last_change":"2020-10-27 04:19:27.889205","last_active":"2020-10-27 04:19:34.143009","last_peered":"2020-10-27 04:19:34.143009","last_clean":"2020-10-27 04:19:34.143009","last_became_active":"2020-10-27 04:19:27.887303","last_became_peered":"2020-10-27 04:19:27.887303","last_unstale":"2020-10-27 04:19:34.143009","last_undegraded":"2020-10-27 04:19:34.143009","last_fullsized":"2020-10-27 04:19:34.143009","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5159749,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3999,"num_write":14,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4789995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"106","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:54.406100","last_change":"2020-10-27 04:29:54.406213","last_active":"2020-10-27 04:29:54.406100","last_peered":"2020-10-27 04:29:54.406100","last_clean":"2020-10-27 04:29:54.406100","last_became_active":"2020-10-27 04:18:45.995583","last_became_peered":"2020-10-27 04:18:45.995583","last_unstale":"2020-10-27 04:29:54.406100","last_undegraded":"2020-10-27 04:29:54.406100","last_fullsized":"2020-10-27 04:29:54.406100","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:54.406082","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:54.406082","last_clean_scrub_stamp":"2020-10-27 04:29:54.406082","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"55'50","reported_seq":"228","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147332","last_change":"2020-10-27 04:19:14.929768","last_active":"2020-10-27 04:19:34.147332","last_peered":"2020-10-27 04:19:34.147332","last_clean":"2020-10-27 04:19:34.147332","last_became_active":"2020-10-27 04:19:14.926647","last_became_peered":"2020-10-27 04:19:14.926647","last_unstale":"2020-10-27 04:19:34.147332","last_undegraded":"2020-10-27 04:19:34.147332","last_fullsized":"2020-10-27 04:19:34.147332","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2302559,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":4243,"num_write":22,"num_write_kb":7245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608346","last_change":"2020-10-27 04:18:55.292214","last_active":"2020-10-27 04:19:34.608346","last_peered":"2020-10-27 04:19:34.608346","last_clean":"2020-10-27 04:19:34.608346","last_became_active":"2020-10-27 04:18:45.992519","last_became_peered":"2020-10-27 04:18:45.992519","last_unstale":"2020-10-27 04:19:34.608346","last_undegraded":"2020-10-27 04:19:34.608346","last_fullsized":"2020-10-27 04:19:34.608346","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.292162","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.292162","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'64","reported_seq":"236","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146895","last_change":"2020-10-27 04:19:14.925901","last_active":"2020-10-27 04:19:34.146895","last_peered":"2020-10-27 04:19:34.146895","last_clean":"2020-10-27 04:19:34.146895","last_became_active":"2020-10-27 04:19:14.922144","last_became_peered":"2020-10-27 04:19:14.922144","last_unstale":"2020-10-27 04:19:34.146895","last_undegraded":"2020-10-27 04:19:34.146895","last_fullsized":"2020-10-27 04:19:34.146895","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6100941,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":68,"num_write_kb":6525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"82","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146863","last_change":"2020-10-27 04:19:14.916148","last_active":"2020-10-27 04:19:34.146863","last_peered":"2020-10-27 04:19:34.146863","last_clean":"2020-10-27 04:19:34.146863","last_became_active":"2020-10-27 04:19:14.914681","last_became_peered":"2020-10-27 04:19:14.914681","last_unstale":"2020-10-27 04:19:34.146863","last_undegraded":"2020-10-27 04:19:34.146863","last_fullsized":"2020-10-27 04:19:34.146863","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:17:46.831755","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:17:46.831755","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'110","reported_seq":"284","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146939","last_change":"2020-10-27 04:19:17.889774","last_active":"2020-10-27 04:19:34.146939","last_peered":"2020-10-27 04:19:34.146939","last_clean":"2020-10-27 04:19:34.146939","last_became_active":"2020-10-27 04:19:17.885352","last_became_peered":"2020-10-27 04:19:17.885352","last_unstale":"2020-10-27 04:19:34.146939","last_undegraded":"2020-10-27 04:19:34.146939","last_fullsized":"2020-10-27 04:19:34.146939","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11563216,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1195,"num_write":102,"num_write_kb":11958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"64'43","reported_seq":"227","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.140212","last_change":"2020-10-27 04:18:55.937872","last_active":"2020-10-27 04:19:34.140212","last_peered":"2020-10-27 04:19:34.140212","last_clean":"2020-10-27 04:19:34.140212","last_became_active":"2020-10-27 04:18:14.271076","last_became_peered":"2020-10-27 04:18:14.271076","last_unstale":"2020-10-27 04:19:34.140212","last_undegraded":"2020-10-27 04:19:34.140212","last_fullsized":"2020-10-27 04:19:34.140212","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6050520,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":8972,"num_write":17,"num_write_kb":7783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"104","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:59.341431","last_change":"2020-10-27 04:29:59.341513","last_active":"2020-10-27 04:29:59.341431","last_peered":"2020-10-27 04:29:59.341431","last_clean":"2020-10-27 04:29:59.341431","last_became_active":"2020-10-27 04:18:45.997645","last_became_peered":"2020-10-27 04:18:45.997645","last_unstale":"2020-10-27 04:29:59.341431","last_undegraded":"2020-10-27 04:29:59.341431","last_fullsized":"2020-10-27 04:29:59.341431","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:59.341421","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:59.341421","last_clean_scrub_stamp":"2020-10-27 04:29:59.341421","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'11","reported_seq":"182","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146908","last_change":"2020-10-27 04:19:14.927715","last_active":"2020-10-27 04:19:34.146908","last_peered":"2020-10-27 04:19:34.146908","last_clean":"2020-10-27 04:19:34.146908","last_became_active":"2020-10-27 04:19:14.921996","last_became_peered":"2020-10-27 04:19:14.921996","last_unstale":"2020-10-27 04:19:34.146908","last_undegraded":"2020-10-27 04:19:34.146908","last_fullsized":"2020-10-27 04:19:34.146908","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"89","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:58.601607","last_change":"2020-10-27 04:29:58.601770","last_active":"2020-10-27 04:29:58.601607","last_peered":"2020-10-27 04:29:58.601607","last_clean":"2020-10-27 04:29:58.601607","last_became_active":"2020-10-27 04:19:27.881858","last_became_peered":"2020-10-27 04:19:27.881858","last_unstale":"2020-10-27 04:29:58.601607","last_undegraded":"2020-10-27 04:29:58.601607","last_fullsized":"2020-10-27 04:29:58.601607","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:58.601579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:58.601579","last_clean_scrub_stamp":"2020-10-27 04:29:58.601579","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"73","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142619","last_change":"2020-10-27 04:19:27.888587","last_active":"2020-10-27 04:19:34.142619","last_peered":"2020-10-27 04:19:34.142619","last_clean":"2020-10-27 04:19:34.142619","last_became_active":"2020-10-27 04:19:27.887602","last_became_peered":"2020-10-27 04:19:27.887602","last_unstale":"2020-10-27 04:19:34.142619","last_undegraded":"2020-10-27 04:19:34.142619","last_fullsized":"2020-10-27 04:19:34.142619","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.119312","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.119312","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"63'111","reported_seq":"286","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602263","last_change":"2020-10-27 04:18:55.939797","last_active":"2020-10-27 04:19:34.602263","last_peered":"2020-10-27 04:19:34.602263","last_clean":"2020-10-27 04:19:34.602263","last_became_active":"2020-10-27 04:18:14.266566","last_became_peered":"2020-10-27 04:18:14.266566","last_unstale":"2020-10-27 04:19:34.602263","last_undegraded":"2020-10-27 04:19:34.602263","last_fullsized":"2020-10-27 04:19:34.602263","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14773353,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":2334,"num_write":144,"num_write_kb":16198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608318","last_change":"2020-10-27 04:19:03.292079","last_active":"2020-10-27 04:19:34.608318","last_peered":"2020-10-27 04:19:34.608318","last_clean":"2020-10-27 04:19:34.608318","last_became_active":"2020-10-27 04:18:53.919537","last_became_peered":"2020-10-27 04:18:53.919537","last_unstale":"2020-10-27 04:19:34.608318","last_undegraded":"2020-10-27 04:19:34.608318","last_fullsized":"2020-10-27 04:19:34.608318","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:03.291964","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:03.291964","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"55'28","reported_seq":"175","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142751","last_change":"2020-10-27 04:19:27.888943","last_active":"2020-10-27 04:19:34.142751","last_peered":"2020-10-27 04:19:34.142751","last_clean":"2020-10-27 04:19:34.142751","last_became_active":"2020-10-27 04:19:27.887163","last_became_peered":"2020-10-27 04:19:27.887163","last_unstale":"2020-10-27 04:19:34.142751","last_undegraded":"2020-10-27 04:19:34.142751","last_fullsized":"2020-10-27 04:19:34.142751","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4218557,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1091,"num_write":16,"num_write_kb":4354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608526","last_change":"2020-10-27 04:19:13.189378","last_active":"2020-10-27 04:19:34.608526","last_peered":"2020-10-27 04:19:34.608526","last_clean":"2020-10-27 04:19:34.608526","last_became_active":"2020-10-27 04:18:53.917040","last_became_peered":"2020-10-27 04:18:53.917040","last_unstale":"2020-10-27 04:19:34.608526","last_undegraded":"2020-10-27 04:19:34.608526","last_fullsized":"2020-10-27 04:19:34.608526","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:13.189245","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:13.189245","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"50'22","reported_seq":"195","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427169","last_change":"2020-10-27 04:18:55.910655","last_active":"2020-10-27 04:19:34.427169","last_peered":"2020-10-27 04:19:34.427169","last_clean":"2020-10-27 04:19:34.427169","last_became_active":"2020-10-27 04:18:14.278718","last_became_peered":"2020-10-27 04:18:14.278718","last_unstale":"2020-10-27 04:19:34.427169","last_undegraded":"2020-10-27 04:19:34.427169","last_fullsized":"2020-10-27 04:19:34.427169","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":44,"num_write_kb":1084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"94","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608286","last_change":"2020-10-27 04:19:01.325786","last_active":"2020-10-27 04:19:34.608286","last_peered":"2020-10-27 04:19:34.608286","last_clean":"2020-10-27 04:19:34.608286","last_became_active":"2020-10-27 04:18:53.915676","last_became_peered":"2020-10-27 04:18:53.915676","last_unstale":"2020-10-27 04:19:34.608286","last_undegraded":"2020-10-27 04:19:34.608286","last_fullsized":"2020-10-27 04:19:34.608286","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:01.325624","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:01.325624","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"64'56","reported_seq":"226","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146963","last_change":"2020-10-27 04:19:17.888785","last_active":"2020-10-27 04:19:34.146963","last_peered":"2020-10-27 04:19:34.146963","last_clean":"2020-10-27 04:19:34.146963","last_became_active":"2020-10-27 04:19:17.885684","last_became_peered":"2020-10-27 04:19:17.885684","last_unstale":"2020-10-27 04:19:34.146963","last_undegraded":"2020-10-27 04:19:34.146963","last_fullsized":"2020-10-27 04:19:34.146963","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2026,"num_write":41,"num_write_kb":5997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":"41","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147288","last_change":"2020-10-27 04:19:17.548013","last_active":"2020-10-27 04:19:34.147288","last_peered":"2020-10-27 04:19:34.147288","last_clean":"2020-10-27 04:19:34.147288","last_became_active":"2020-10-27 04:19:14.920337","last_became_peered":"2020-10-27 04:19:14.920337","last_unstale":"2020-10-27 04:19:34.147288","last_undegraded":"2020-10-27 04:19:34.147288","last_fullsized":"2020-10-27 04:19:34.147288","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:17.547918","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:17.547918","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"50'26","reported_seq":"202","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427394","last_change":"2020-10-27 04:18:55.909529","last_active":"2020-10-27 04:19:34.427394","last_peered":"2020-10-27 04:19:34.427394","last_clean":"2020-10-27 04:19:34.427394","last_became_active":"2020-10-27 04:18:14.273127","last_became_peered":"2020-10-27 04:18:14.273127","last_unstale":"2020-10-27 04:19:34.427394","last_undegraded":"2020-10-27 04:19:34.427394","last_fullsized":"2020-10-27 04:19:34.427394","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":547,"num_write":10,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608636","last_change":"2020-10-27 04:19:07.322883","last_active":"2020-10-27 04:19:34.608636","last_peered":"2020-10-27 04:19:34.608636","last_clean":"2020-10-27 04:19:34.608636","last_became_active":"2020-10-27 04:18:53.920956","last_became_peered":"2020-10-27 04:18:53.920956","last_unstale":"2020-10-27 04:19:34.608636","last_undegraded":"2020-10-27 04:19:34.608636","last_fullsized":"2020-10-27 04:19:34.608636","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:07.322732","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:19:07.322732","last_clean_scrub_stamp":"2020-10-27 04:19:07.322732","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"55'62","reported_seq":"242","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.604887","last_change":"2020-10-27 04:18:55.917852","last_active":"2020-10-27 04:19:34.604887","last_peered":"2020-10-27 04:19:34.604887","last_clean":"2020-10-27 04:19:34.604887","last_became_active":"2020-10-27 04:18:14.266855","last_became_peered":"2020-10-27 04:18:14.266855","last_unstale":"2020-10-27 04:19:34.604887","last_undegraded":"2020-10-27 04:19:34.604887","last_fullsized":"2020-10-27 04:19:34.604887","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201717,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":7644,"num_write":22,"num_write_kb":3533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145244160,"data_stored":176377794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575026,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1060788,"kb_used_data":12204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93311052,"statfs":{"total":96636764160,"available":95550517248,"internally_reserved":1073741824,"allocated":12496896,"data_stored":12292194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:15 2020","interfaces":[{"interface":"back","average":{"1min":49.296999999999997,"5min":12.644,"15min":5.9509999999999996},"min":{"1min":1.7649999999999999,"5min":1.2589999999999999,"15min":1.218},"max":{"1min":985.40999999999997,"5min":985.40999999999997,"15min":985.40999999999997},"last":2.218},{"interface":"front","average":{"1min":2.452,"5min":2.4180000000000001,"15min":10.823},"min":{"1min":1.369,"5min":1.2030000000000001,"15min":1.1140000000000001},"max":{"1min":3.79,"5min":3.9300000000000002,"15min":971.95100000000002},"last":2.8319999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":11.887,"15min":8.0559999999999992},"min":{"1min":1.196,"5min":1.196,"15min":1.139},"max":{"1min":3.0030000000000001,"5min":834.52300000000002,"15min":834.52300000000002},"last":1.782},{"interface":"front","average":{"1min":2.6219999999999999,"5min":10.395,"15min":7.609},"min":{"1min":2.1320000000000001,"5min":1.589,"15min":1.095},"max":{"1min":3.218,"5min":621.11199999999997,"15min":675.13099999999997},"last":2.2949999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":37.174999999999997,"5min":9.423,"15min":10.127000000000001},"min":{"1min":1.8260000000000001,"5min":1.5900000000000001,"15min":1.2649999999999999},"max":{"1min":595.22699999999998,"5min":595.22699999999998,"15min":749.68899999999996},"last":2.673},{"interface":"front","average":{"1min":26.971,"5min":7.2380000000000004,"15min":7.1200000000000001},"min":{"1min":1.7769999999999999,"5min":1.5860000000000001,"15min":1.1799999999999999},"max":{"1min":423.61200000000002,"5min":423.61200000000002,"15min":503.75799999999998},"last":1.861}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.7709999999999999,"5min":10.558,"15min":5.4829999999999997},"min":{"1min":2.0529999999999999,"5min":1.9239999999999999,"15min":1.6120000000000001},"max":{"1min":3.29,"5min":810.279,"15min":810.279},"last":2.262},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.6219999999999999,"15min":19.773},"min":{"1min":1.694,"5min":1.6879999999999999,"15min":1.552},"max":{"1min":3.1059999999999999,"5min":3.859,"15min":897.61800000000005},"last":2.5640000000000001}]},{"osd":4,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8399999999999999,"5min":22.492999999999999,"15min":13.928000000000001},"min":{"1min":2.3260000000000001,"5min":1.845,"15min":1.5569999999999999},"max":{"1min":3.2559999999999998,"5min":671.45899999999995,"15min":671.45899999999995},"last":2.887},{"interface":"front","average":{"1min":2.706,"5min":11.789999999999999,"15min":9.1460000000000008},"min":{"1min":1.931,"5min":1.931,"15min":1.665},"max":{"1min":3.2000000000000002,"5min":639.77499999999998,"15min":639.77499999999998},"last":2.758}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":4.444,"15min":5.7809999999999997},"min":{"1min":1.8460000000000001,"5min":1.7,"15min":1.7},"max":{"1min":3.2709999999999999,"5min":159.59399999999999,"15min":735.70500000000004},"last":3.0699999999999998},{"interface":"front","average":{"1min":3.0550000000000002,"5min":16.914999999999999,"15min":16.773},"min":{"1min":2.343,"5min":2.343,"15min":2.1909999999999998},"max":{"1min":3.6560000000000001,"5min":923.20100000000002,"15min":923.20100000000002},"last":2.343}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.3700000000000001,"5min":15.603,"15min":10.483000000000001},"min":{"1min":2.581,"5min":2.581,"15min":2.2679999999999998},"max":{"1min":5.2469999999999999,"5min":992.976,"15min":992.976},"last":679.41899999999998},{"interface":"front","average":{"1min":3.0049999999999999,"5min":6.5810000000000004,"15min":4.5540000000000003},"min":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":1.9530000000000001},"max":{"1min":3.8420000000000001,"5min":390.29300000000001,"15min":390.29300000000001},"last":3.1309999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575025,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1067020,"kb_used_data":18436,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93304820,"statfs":{"total":96636764160,"available":95544135680,"internally_reserved":1073741824,"allocated":18878464,"data_stored":23929015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":2.387,"15min":2.395},"min":{"1min":1.641,"5min":1.571,"15min":1.022},"max":{"1min":3.0219999999999998,"5min":3.4750000000000001,"15min":4.4809999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.5310000000000001,"15min":2.5409999999999999},"min":{"1min":1.53,"5min":1.53,"15min":1.083},"max":{"1min":3.5670000000000002,"5min":3.794,"15min":7.4249999999999998},"last":3.0099999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.109,"15min":2.1930000000000001},"min":{"1min":1.198,"5min":1.175,"15min":1.115},"max":{"1min":2.7810000000000001,"5min":3.2370000000000001,"15min":10.25},"last":2.8330000000000002},{"interface":"front","average":{"1min":2.7709999999999999,"5min":15.755000000000001,"15min":6.71},"min":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.0549999999999999},"max":{"1min":3.649,"5min":1340.4169999999999,"15min":1340.4169999999999},"last":1.724}]},{"osd":2,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.5190000000000001,"5min":2.597,"15min":9.9090000000000007},"min":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.452},"max":{"1min":3.1579999999999999,"5min":3.621,"15min":900.75599999999997},"last":2.367},{"interface":"front","average":{"1min":3.2090000000000001,"5min":2.8330000000000002,"15min":2.7799999999999998},"min":{"1min":2.3420000000000001,"5min":1.605,"15min":1.452},"max":{"1min":4.133,"5min":4.133,"15min":4.7000000000000002},"last":2.7690000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:40 2020","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.8479999999999999,"15min":7.6399999999999997},"min":{"1min":2.0139999999999998,"5min":1.7370000000000001,"15min":1.607},"max":{"1min":3.6840000000000002,"5min":4.452,"15min":637.64599999999996},"last":3.407},{"interface":"front","average":{"1min":3.0979999999999999,"5min":12.949999999999999,"15min":8.9559999999999995},"min":{"1min":2.2290000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"max":{"1min":3.8940000000000001,"5min":1002.468,"15min":1002.468},"last":2.895}]},{"osd":4,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4630000000000001,"5min":2.589,"15min":10.563000000000001},"min":{"1min":1.6459999999999999,"5min":1.462,"15min":1.452},"max":{"1min":3.0089999999999999,"5min":3.7280000000000002,"15min":888.96900000000005},"last":1.79},{"interface":"front","average":{"1min":2.835,"5min":9.891,"15min":7.2279999999999998},"min":{"1min":1.498,"5min":1.498,"15min":1.498},"max":{"1min":3.802,"5min":700.48099999999999,"15min":700.48099999999999},"last":3.1930000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.9039999999999999,"5min":2.9630000000000001,"15min":5.8529999999999998},"min":{"1min":2.2519999999999998,"5min":1.506,"15min":1.506},"max":{"1min":3.7200000000000002,"5min":4.3019999999999996,"15min":552.64099999999996},"last":3.1160000000000001},{"interface":"front","average":{"1min":3.3159999999999998,"5min":12.209,"15min":6.0279999999999996},"min":{"1min":2.3119999999999998,"5min":1.99,"15min":1.488},"max":{"1min":4.0350000000000001,"5min":912.10799999999995,"15min":912.10799999999995},"last":3.355}]},{"osd":7,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":3.0499999999999998,"5min":23.687000000000001,"15min":11.260999999999999},"min":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":1.52},"max":{"1min":3.786,"5min":787.13199999999995,"15min":787.13199999999995},"last":3.2789999999999999},{"interface":"front","average":{"1min":3.569,"5min":3.2679999999999998,"15min":9.1240000000000006},"min":{"1min":2.871,"5min":2.492,"15min":1.966},"max":{"1min":4.6680000000000001,"5min":4.6680000000000001,"15min":1000.52},"last":2.6899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575025,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1068864,"kb_used_data":20280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302976,"statfs":{"total":96636764160,"available":95542247424,"internally_reserved":1073741824,"allocated":20766720,"data_stored":26346047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.6120000000000001,"5min":2.4689999999999999,"15min":6.6299999999999999},"min":{"1min":1.3720000000000001,"5min":1.302,"15min":1.0449999999999999},"max":{"1min":3.3140000000000001,"5min":3.8340000000000001,"15min":787.62400000000002},"last":2.7050000000000001},{"interface":"front","average":{"1min":2.2050000000000001,"5min":12.976000000000001,"15min":14.279999999999999},"min":{"1min":1.4490000000000001,"5min":1.226,"15min":0.89600000000000002},"max":{"1min":3.1429999999999998,"5min":910.65099999999995,"15min":910.65099999999995},"last":2.4209999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":2.5209999999999999,"15min":4.1239999999999997},"min":{"1min":1.591,"5min":1.474,"15min":1.194},"max":{"1min":3.411,"5min":3.8730000000000002,"15min":484.69600000000003},"last":785.25099999999998},{"interface":"front","average":{"1min":2.387,"5min":11.192,"15min":18.960000000000001},"min":{"1min":1.522,"5min":1.522,"15min":1.3979999999999999},"max":{"1min":2.887,"5min":834.95000000000005,"15min":877.15300000000002},"last":3.8279999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":19.675999999999998,"5min":6.1299999999999999,"15min":3.7130000000000001},"min":{"1min":1.972,"5min":1.52,"15min":1.2509999999999999},"max":{"1min":323.01799999999997,"5min":323.01799999999997,"15min":323.01799999999997},"last":2.7610000000000001},{"interface":"front","average":{"1min":2.5329999999999999,"5min":8.093,"15min":6.774},"min":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.238},"max":{"1min":4.3150000000000004,"5min":599.65599999999995,"15min":599.65599999999995},"last":3.9020000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":2.8809999999999998,"15min":2.996},"min":{"1min":1.6950000000000001,"5min":1.661,"15min":1.661},"max":{"1min":3.8069999999999999,"5min":3.899,"15min":4.6299999999999999},"last":2.798},{"interface":"front","average":{"1min":3.0310000000000001,"5min":4.9000000000000004,"15min":4.4480000000000004},"min":{"1min":2.0419999999999998,"5min":1.7190000000000001,"15min":1.3220000000000001},"max":{"1min":4.4039999999999999,"5min":218.10400000000001,"15min":228.80099999999999},"last":3.9889999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.1080000000000001,"5min":3.0630000000000002,"15min":2.968},"min":{"1min":2.399,"5min":2.2349999999999999,"15min":1.5149999999999999},"max":{"1min":3.9729999999999999,"5min":4.1669999999999998,"15min":4.633},"last":3.0600000000000001},{"interface":"front","average":{"1min":2.895,"5min":2.8479999999999999,"15min":5.8499999999999996},"min":{"1min":1.8220000000000001,"5min":1.655,"15min":1.615},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":623.83100000000002},"last":4.0570000000000004}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.2050000000000001,"5min":9.7910000000000004,"15min":10.93},"min":{"1min":2.4510000000000001,"5min":2.2799999999999998,"15min":1.7110000000000001},"max":{"1min":3.895,"5min":727.97400000000005,"15min":816.51300000000003},"last":2.835},{"interface":"front","average":{"1min":2.8940000000000001,"5min":18.178999999999998,"15min":12.436},"min":{"1min":2.2229999999999999,"5min":1.8320000000000001,"15min":1.5700000000000001},"max":{"1min":3.359,"5min":994.68299999999999,"15min":994.68299999999999},"last":2.4940000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.8940000000000001,"5min":6.3639999999999999,"15min":8.2070000000000007},"min":{"1min":2.3940000000000001,"5min":2.0190000000000001,"15min":1.6919999999999999},"max":{"1min":3.496,"5min":346.387,"15min":504.964},"last":2.871},{"interface":"front","average":{"1min":2.988,"5min":3.081,"15min":13.677},"min":{"1min":2.121,"5min":1.9299999999999999,"15min":1.2769999999999999},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":891.01599999999996},"last":4.125}]}]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064972,"kb_used_data":16388,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306868,"statfs":{"total":96636764160,"available":95546232832,"internally_reserved":1073741824,"allocated":16781312,"data_stored":20017524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":14.038,"5min":4.6779999999999999,"15min":3.419},"min":{"1min":1.141,"5min":1.141,"15min":1.054},"max":{"1min":261.02499999999998,"5min":261.02499999999998,"15min":261.02499999999998},"last":1.633},{"interface":"front","average":{"1min":2.7450000000000001,"5min":2.512,"15min":17.265999999999998},"min":{"1min":1.3899999999999999,"5min":1.234,"15min":1.1419999999999999},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":808.98800000000006},"last":2.1509999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":6.6239999999999997,"15min":6.2779999999999996},"min":{"1min":1.702,"5min":1.4510000000000001,"15min":1.0980000000000001},"max":{"1min":4.2809999999999997,"5min":454.637,"15min":654.36699999999996},"last":3.218},{"interface":"front","average":{"1min":3.145,"5min":2.6920000000000002,"15min":5.0830000000000002},"min":{"1min":1.8420000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":949.18700000000001},"last":2.2549999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.351,"5min":7.1449999999999996,"15min":12.432},"min":{"1min":1.4350000000000001,"5min":1.4219999999999999,"15min":1.165},"max":{"1min":5.0810000000000004,"5min":451.41800000000001,"15min":976.43700000000001},"last":2.956},{"interface":"front","average":{"1min":2.6200000000000001,"5min":2.6419999999999999,"15min":11.617000000000001},"min":{"1min":1.754,"5min":1.754,"15min":1.623},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":780.63900000000001},"last":3.3769999999999998}]},{"osd":3,"last update":"Tue Oct 27 04:31:33 2020","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.6269999999999998,"15min":5.4489999999999998},"min":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.3759999999999999},"max":{"1min":3.6629999999999998,"5min":4.8710000000000004,"15min":478.51299999999998},"last":3.5529999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":6.0709999999999997,"15min":3.8820000000000001},"min":{"1min":2.2709999999999999,"5min":1.75,"15min":1.593},"max":{"1min":4.3310000000000004,"5min":344.82600000000002,"15min":344.82600000000002},"last":3.6400000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":13.503,"15min":7.8840000000000003},"min":{"1min":1.9970000000000001,"5min":1.9730000000000001,"15min":1.3819999999999999},"max":{"1min":4.0209999999999999,"5min":800.56899999999996,"15min":800.56899999999996},"last":3.3140000000000001},{"interface":"front","average":{"1min":19.757999999999999,"5min":14.157,"15min":7.9299999999999997},"min":{"1min":2.2829999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"max":{"1min":286.78800000000001,"5min":795.65599999999995,"15min":795.65599999999995},"last":3.5939999999999999}]},{"osd":6,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8820000000000001,"5min":2.7810000000000001,"15min":7.1779999999999999},"min":{"1min":2.0089999999999999,"5min":1.583,"15min":1.583},"max":{"1min":5.2489999999999997,"5min":5.8540000000000001,"15min":872.47000000000003},"last":3.036},{"interface":"front","average":{"1min":3.0329999999999999,"5min":3.0579999999999998,"15min":3.0230000000000001},"min":{"1min":2.089,"5min":2.089,"15min":1.298},"max":{"1min":3.851,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":3.4820000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":6.2850000000000001,"15min":7.04},"min":{"1min":2.194,"5min":1.9630000000000001,"15min":1.9630000000000001},"max":{"1min":5.2960000000000003,"5min":323.93400000000003,"15min":882.08600000000001},"last":3.4300000000000002},{"interface":"front","average":{"1min":3.625,"5min":3.5209999999999999,"15min":6.3170000000000002},"min":{"1min":2.883,"5min":2.2669999999999999,"15min":1.8460000000000001},"max":{"1min":4.3700000000000001,"5min":26.504000000000001,"15min":926.90700000000004},"last":3.6760000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073392,"kb_used_data":24808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93298448,"statfs":{"total":96636764160,"available":95537610752,"internally_reserved":1073741824,"allocated":25403392,"data_stored":30154857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":7.9109999999999996,"15min":4.1260000000000003},"min":{"1min":1.466,"5min":1.1080000000000001,"15min":0.90600000000000003},"max":{"1min":3.4100000000000001,"5min":545.69000000000005,"15min":545.69000000000005},"last":1.8220000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":16.379000000000001,"15min":7.0330000000000004},"min":{"1min":1.6259999999999999,"5min":1.145,"15min":0.80900000000000005},"max":{"1min":3.7789999999999999,"5min":658.68899999999996,"15min":658.68899999999996},"last":2.7639999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":4.7110000000000003,"15min":2.8610000000000002},"min":{"1min":1.3440000000000001,"5min":1.212,"15min":0.76900000000000002},"max":{"1min":3.4079999999999999,"5min":272.61700000000002,"15min":272.61700000000002},"last":2.2610000000000001},{"interface":"front","average":{"1min":2.7120000000000002,"5min":6.8280000000000003,"15min":5.0869999999999997},"min":{"1min":1.893,"5min":1.141,"15min":0.97099999999999997},"max":{"1min":3.718,"5min":251.756,"15min":251.756},"last":2.879}]},{"osd":3,"last update":"Tue Oct 27 04:31:38 2020","interfaces":[{"interface":"back","average":{"1min":60.173000000000002,"5min":13.944000000000001,"15min":13.381},"min":{"1min":1.9079999999999999,"5min":1.365,"15min":1.2030000000000001},"max":{"1min":866.04100000000005,"5min":866.04100000000005,"15min":923.22299999999996},"last":1.7829999999999999},{"interface":"front","average":{"1min":2.7330000000000001,"5min":14.502000000000001,"15min":8.968},"min":{"1min":1.8740000000000001,"5min":1.4970000000000001,"15min":1.26},"max":{"1min":3.5369999999999999,"5min":796.69100000000003,"15min":796.69100000000003},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":42.18,"5min":10.504,"15min":13.028},"min":{"1min":2.0569999999999999,"5min":1.167,"15min":1.167},"max":{"1min":712.15599999999995,"5min":712.15599999999995,"15min":809.928},"last":2.8279999999999998},{"interface":"front","average":{"1min":2.6699999999999999,"5min":2.5510000000000002,"15min":6.2770000000000001},"min":{"1min":1.8460000000000001,"5min":1.401,"15min":1.1379999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":636.61699999999996},"last":2.9830000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":2.593,"15min":6.6749999999999998},"min":{"1min":1.9430000000000001,"5min":1.667,"15min":1.4079999999999999},"max":{"1min":3.7309999999999999,"5min":4.9260000000000002,"15min":577.61300000000006},"last":2.9350000000000001},{"interface":"front","average":{"1min":56.189,"5min":21.494,"15min":9.8610000000000007},"min":{"1min":1.946,"5min":1.5449999999999999,"15min":1.5449999999999999},"max":{"1min":961.779,"5min":961.779,"15min":961.779},"last":3.0800000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.8999999999999999,"5min":9.3569999999999993,"15min":12.731999999999999},"min":{"1min":1.988,"5min":1.8859999999999999,"15min":1.276},"max":{"1min":3.512,"5min":503.51400000000001,"15min":824.16200000000003},"last":3.4289999999999998},{"interface":"front","average":{"1min":3.1749999999999998,"5min":3.1589999999999998,"15min":6.5590000000000002},"min":{"1min":2.4260000000000002,"5min":1.702,"15min":1.5600000000000001},"max":{"1min":3.8519999999999999,"5min":4.6280000000000001,"15min":906.19299999999998},"last":3.871}]},{"osd":7,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":2.8140000000000001,"15min":3.0979999999999999},"min":{"1min":2.222,"5min":1.73,"15min":1.5760000000000001},"max":{"1min":3.77,"5min":4.1669999999999998,"15min":12.324},"last":3.3690000000000002},{"interface":"front","average":{"1min":33.110999999999997,"5min":9.2379999999999995,"15min":6.2720000000000002},"min":{"1min":2.6469999999999998,"5min":1.9750000000000001,"15min":1.4390000000000001},"max":{"1min":510.59899999999999,"5min":510.59899999999999,"15min":510.59899999999999},"last":3.9260000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1065520,"kb_used_data":16936,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306320,"statfs":{"total":96636764160,"available":95545671680,"internally_reserved":1073741824,"allocated":17342464,"data_stored":24243891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.8640000000000001,"15min":2.7629999999999999},"min":{"1min":1.603,"5min":1.226,"15min":0.72199999999999998},"max":{"1min":2.8530000000000002,"5min":2.8530000000000002,"15min":194.143},"last":1.677},{"interface":"front","average":{"1min":2.6930000000000001,"5min":11.317,"15min":10.750999999999999},"min":{"1min":1.9590000000000001,"5min":1.353,"15min":0.81299999999999994},"max":{"1min":4.0609999999999999,"5min":656.79600000000005,"15min":731.65800000000002},"last":2.3330000000000002}]},{"osd":2,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":12.202,"15min":12.997},"min":{"1min":1.5920000000000001,"5min":1.3089999999999999,"15min":0.70199999999999996},"max":{"1min":2.895,"5min":992.125,"15min":992.125},"last":1.7569999999999999},{"interface":"front","average":{"1min":2.7709999999999999,"5min":9.4710000000000001,"15min":5.9850000000000003},"min":{"1min":1.52,"5min":1.3640000000000001,"15min":0.68500000000000005},"max":{"1min":3.6920000000000002,"5min":727.048,"15min":727.048},"last":2.5110000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:36 2020","interfaces":[{"interface":"back","average":{"1min":5.1799999999999997,"5min":3.4060000000000001,"15min":18.274000000000001},"min":{"1min":1.4119999999999999,"5min":1.3720000000000001,"15min":0.92400000000000004},"max":{"1min":61.938000000000002,"5min":63.479999999999997,"15min":879.947},"last":2.9489999999999998},{"interface":"front","average":{"1min":2.6440000000000001,"5min":9.3989999999999991,"15min":8.5289999999999999},"min":{"1min":1.6319999999999999,"5min":1.595,"15min":1.1379999999999999},"max":{"1min":3.4289999999999998,"5min":466.53800000000001,"15min":466.53800000000001},"last":2.2589999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":2.4849999999999999,"15min":3.8580000000000001},"min":{"1min":1.6579999999999999,"5min":1.464,"15min":1.224},"max":{"1min":3.7919999999999998,"5min":3.7919999999999998,"15min":325.25},"last":2.5790000000000002},{"interface":"front","average":{"1min":3.335,"5min":12.461,"15min":6.266},"min":{"1min":2.4980000000000002,"5min":2.121,"15min":1.052},"max":{"1min":4.5529999999999999,"5min":954.96299999999997,"15min":954.96299999999997},"last":3.266}]},{"osd":5,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.5819999999999999,"15min":4.1840000000000002},"min":{"1min":1.675,"5min":1.5349999999999999,"15min":1.173},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":436.06900000000002},"last":3.9769999999999999},{"interface":"front","average":{"1min":3.1739999999999999,"5min":6.9080000000000004,"15min":4.6369999999999996},"min":{"1min":2.0840000000000001,"5min":1.8919999999999999,"15min":1.1990000000000001},"max":{"1min":4.3899999999999997,"5min":425.92899999999997,"15min":425.92899999999997},"last":2.4180000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.96,"5min":2.7599999999999998,"15min":5.3079999999999998},"min":{"1min":2.0219999999999998,"5min":1.855,"15min":1.486},"max":{"1min":3.919,"5min":4.306,"15min":431.327},"last":3.919},{"interface":"front","average":{"1min":3.4300000000000002,"5min":3.2599999999999998,"15min":6.3230000000000004},"min":{"1min":2.5299999999999998,"5min":2.2530000000000001,"15min":1.3879999999999999},"max":{"1min":4.476,"5min":4.476,"15min":800.98800000000006},"last":3.2229999999999999}]},{"osd":7,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":6.6070000000000002,"15min":12.765000000000001},"min":{"1min":2.1960000000000002,"5min":1.976,"15min":1.2789999999999999},"max":{"1min":3.8100000000000001,"5min":448.084,"15min":970.05899999999997},"last":2.4580000000000002},{"interface":"front","average":{"1min":3.1030000000000002,"5min":2.9580000000000002,"15min":5.1280000000000001},"min":{"1min":1.913,"5min":1.913,"15min":1.609},"max":{"1min":4.016,"5min":4.3490000000000002,"15min":498.803},"last":4.016}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064984,"kb_used_data":16400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306856,"statfs":{"total":96636764160,"available":95546220544,"internally_reserved":1073741824,"allocated":16793600,"data_stored":16954654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":53.104999999999997,"5min":13.978999999999999,"15min":11.644},"min":{"1min":1.3340000000000001,"5min":1.1830000000000001,"15min":1.1200000000000001},"max":{"1min":924.39700000000005,"5min":924.39700000000005,"15min":924.39700000000005},"last":3.032},{"interface":"front","average":{"1min":2.355,"5min":2.2330000000000001,"15min":6.7709999999999999},"min":{"1min":1.52,"5min":0.879,"15min":0.68500000000000005},"max":{"1min":2.984,"5min":5.2169999999999996,"15min":948.09299999999996},"last":1.7849999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":14.672000000000001,"15min":6.3040000000000003},"min":{"1min":1.569,"5min":1.123,"15min":0.84699999999999998},"max":{"1min":3.335,"5min":594.51700000000005,"15min":594.51700000000005},"last":1.6739999999999999},{"interface":"front","average":{"1min":2.5619999999999998,"5min":4.0250000000000004,"15min":6.5709999999999997},"min":{"1min":1.7549999999999999,"5min":1.167,"15min":0.95299999999999996},"max":{"1min":3.0720000000000001,"5min":158.51599999999999,"15min":879.13699999999994},"last":2.4180000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:28 2020","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":2.4409999999999998,"15min":5.0830000000000002},"min":{"1min":1.635,"5min":1.2729999999999999,"15min":1.2729999999999999},"max":{"1min":3.4830000000000001,"5min":3.6869999999999998,"15min":615.34500000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":55.475999999999999,"5min":13.064,"15min":6.0110000000000001},"min":{"1min":1.79,"5min":1.1930000000000001,"15min":1.1930000000000001},"max":{"1min":902.85299999999995,"5min":902.85299999999995,"15min":902.85299999999995},"last":2.3679999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.2090000000000001,"5min":2.359,"15min":4.774},"min":{"1min":1.732,"5min":1.54,"15min":1.329},"max":{"1min":3.3359999999999999,"5min":3.766,"15min":551.82500000000005},"last":3.9729999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":19.233000000000001,"15min":14.946},"min":{"1min":1.7669999999999999,"5min":1.7470000000000001,"15min":1.248},"max":{"1min":4.3780000000000001,"5min":953.96799999999996,"15min":953.96799999999996},"last":2.4929999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9790000000000001,"5min":8.7840000000000007,"15min":15.339},"min":{"1min":1.986,"5min":1.7410000000000001,"15min":1.282},"max":{"1min":4.476,"5min":504.077,"15min":975.08900000000006},"last":2.5739999999999998},{"interface":"front","average":{"1min":3.0169999999999999,"5min":12.907999999999999,"15min":11.02},"min":{"1min":2,"5min":1.6439999999999999,"15min":1.518},"max":{"1min":3.9870000000000001,"5min":504.51799999999997,"15min":504.51799999999997},"last":2.774}]},{"osd":6,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":2.7069999999999999,"15min":8.0600000000000005},"min":{"1min":1.821,"5min":1.6890000000000001,"15min":1.506},"max":{"1min":3.427,"5min":3.871,"15min":872.27499999999998},"last":4.04},{"interface":"front","average":{"1min":3.1259999999999999,"5min":3.044,"15min":2.9540000000000002},"min":{"1min":2.2679999999999998,"5min":1.9419999999999999,"15min":1.669},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.5350000000000001}]},{"osd":7,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":10.723000000000001,"15min":6.4370000000000003},"min":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":1.7450000000000001},"max":{"1min":3.891,"5min":510.29599999999999,"15min":510.29599999999999},"last":4.0899999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.1699999999999999,"15min":9.0950000000000006},"min":{"1min":2.706,"5min":2.2109999999999999,"15min":1.504},"max":{"1min":3.831,"5min":5.2839999999999998,"15min":547.11300000000006},"last":2.8210000000000002}]}]},{"osd":3,"up_from":69,"seq":296352743582,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073196,"kb_used_data":24612,"kb_used_omap":7,"kb_used_meta":1048568,"kb_avail":93298644,"statfs":{"total":96636764160,"available":95537811456,"internally_reserved":1073741824,"allocated":25202688,"data_stored":26366732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":1073733794},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":8.8119999999999994,"15min":11.82},"min":{"1min":1.286,"5min":1.014,"15min":1.014},"max":{"1min":2.7869999999999999,"5min":706.96100000000001,"15min":938.43399999999997},"last":2.2669999999999999},{"interface":"front","average":{"1min":44.875999999999998,"5min":19.175000000000001,"15min":7.8029999999999999},"min":{"1min":1.446,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":896.04899999999998,"5min":896.04899999999998,"15min":896.04899999999998},"last":1.915}]},{"osd":1,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":22.134,"5min":6.2039999999999997,"15min":3.9359999999999999},"min":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.069},"max":{"1min":423.48000000000002,"5min":423.48000000000002,"15min":423.48000000000002},"last":2.399},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.3679999999999999,"15min":5.79},"min":{"1min":1.5089999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":3.3540000000000001,"5min":3.472,"15min":678.43899999999996},"last":2.931}]},{"osd":2,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.5699999999999998,"5min":2.4590000000000001,"15min":5.2720000000000002},"min":{"1min":1.6379999999999999,"5min":1.3620000000000001,"15min":1.194},"max":{"1min":3.4199999999999999,"5min":3.7879999999999998,"15min":971.47500000000002},"last":613.66600000000005},{"interface":"front","average":{"1min":5.5800000000000001,"5min":3.29,"15min":2.867},"min":{"1min":2.1019999999999999,"5min":1.25,"15min":1.143},"max":{"1min":63.442,"5min":63.442,"15min":63.442},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":31.300999999999998,"5min":16.239999999999998,"15min":12.332000000000001},"min":{"1min":1.9990000000000001,"5min":1.381,"15min":1.381},"max":{"1min":604.30799999999999,"5min":792.40700000000004,"15min":955.47699999999998},"last":2.468},{"interface":"front","average":{"1min":2.5899999999999999,"5min":4.0339999999999998,"15min":3.0310000000000001},"min":{"1min":1.738,"5min":1.073,"15min":1.073},"max":{"1min":3.556,"5min":151.703,"15min":151.703},"last":2.6699999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":36.460999999999999,"5min":9.4410000000000007,"15min":7.3220000000000001},"min":{"1min":1.8859999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"max":{"1min":710.23599999999999,"5min":710.23599999999999,"15min":710.23599999999999},"last":2.141},{"interface":"front","average":{"1min":15.246,"5min":5.4690000000000003,"15min":7.5579999999999998},"min":{"1min":1.962,"5min":1.554,"15min":1.554},"max":{"1min":261.73599999999999,"5min":261.73599999999999,"15min":607.42600000000004},"last":2.9740000000000002}]},{"osd":6,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":28.733000000000001,"5min":13.307,"15min":6.734},"min":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.55},"max":{"1min":522.255,"5min":522.255,"15min":522.255},"last":2.3610000000000002},{"interface":"front","average":{"1min":2.9729999999999999,"5min":3.0019999999999998,"15min":2.9929999999999999},"min":{"1min":2.2719999999999998,"5min":1.772,"15min":1.772},"max":{"1min":3.8439999999999999,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":3.0339999999999998}]},{"osd":7,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":12.74,"15min":8.8989999999999991},"min":{"1min":2.1600000000000001,"5min":2.0979999999999999,"15min":1.667},"max":{"1min":3.6429999999999998,"5min":607.92200000000003,"15min":867.16300000000001},"last":2.617},{"interface":"front","average":{"1min":3.0920000000000001,"5min":16.859999999999999,"15min":9.5329999999999995},"min":{"1min":2.0529999999999999,"5min":1.5960000000000001,"15min":1.3100000000000001},"max":{"1min":3.8279999999999998,"5min":746.22299999999996,"15min":746.22299999999996},"last":2.5640000000000001}]}]}]}} Ignoring malparsed backtrace: instructing pg 2.ds0 on osd.5 to deep-scrub , 2020-10-27T04:32:32.200 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:32:52.202 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.663 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.771 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:18:52.955136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=52, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:41.487 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-10-27T04:36:41.488 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:36:41.522 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: 2020-10-27T04:36:58.697 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1027/qa/tasks/ceph.py", line 1873, in task Ignoring malparsed backtrace: 2020-10-27T04:37:00.234 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-27T04:37:00.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com , 2020-10-27T04:37:00.235 INFO:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Ignoring malparsed backtrace: ceph version 14.2.11-93-g1a7881c1ef (1a7881c1ef0c8e42f605de3c2b81c24c450aa46c) nautilus (stable) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7f173b3864f4] , 2: (ReadOp::_finish(TestOp::CallbackInfo*)+0x1292) [0x55e87ba78562] Ignoring malparsed backtrace: 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:18.424908", , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.589 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:32.937349", , 2020-10-27T04:18:58.603 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" Ignoring malparsed backtrace: 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.040020", , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "throttled" , 2020-10-27T04:18:58.617 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.630 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "done" , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: } , 2020-10-27T04:18:58.631 INFO:teuthology.orchestra.run.smithi037.stdout: ] Ignoring malparsed backtrace: 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.644 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:38.044446", Ignoring malparsed backtrace: 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "queued_for_pg" , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.658 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:44.024020", , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "sub_op_started" , 2020-10-27T04:18:58.671 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.684 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.685 INFO:teuthology.orchestra.run.smithi037.stdout: { Ignoring malparsed backtrace: 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: }, , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.698 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:53.982234", Ignoring malparsed backtrace: 2020-10-27T04:18:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:54.002520", , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "initiated" , 2020-10-27T04:18:58.712 INFO:teuthology.orchestra.run.smithi037.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client": "client.4294", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "client_addr": "172.21.15.39:0/1113100290", , 2020-10-27T04:18:58.725 INFO:teuthology.orchestra.run.smithi037.stdout: "tid": 673 Ignoring malparsed backtrace: 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: { , 2020-10-27T04:18:58.738 INFO:teuthology.orchestra.run.smithi037.stdout: "time": "2020-10-27 04:18:56.931035", , 2020-10-27T04:18:58.739 INFO:teuthology.orchestra.run.smithi037.stdout: "event": "reached_pg" Ignoring malparsed backtrace: 2020-10-27T04:19:31.121 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd reweight 7 1 , reweighted osd.7 to 1 (10000) , 2020-10-27T04:19:32.180 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd primary-affinity 7 1 Ignoring malparsed backtrace: 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap-prune", , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: "nautilus" , 2020-10-27T04:19:47.029 INFO:teuthology.orchestra.run.smithi039.stdout: ], Ignoring malparsed backtrace: 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "mgrmap": { , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 3, , 2020-10-27T04:19:47.044 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 4104, Ignoring malparsed backtrace: 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", , 2020-10-27T04:19:47.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.072 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", , 2020-10-27T04:19:47.086 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", , 2020-10-27T04:19:47.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", , 2020-10-27T04:19:47.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.158 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", Ignoring malparsed backtrace: 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.199 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.213 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", , 2020-10-27T04:19:47.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", Ignoring malparsed backtrace: 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.268 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to persist completed events", , 2020-10-27T04:19:47.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.295 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", , 2020-10-27T04:19:47.296 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": {} Ignoring malparsed backtrace: 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.309 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.323 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { Ignoring malparsed backtrace: 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.364 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", , 2020-10-27T04:19:47.365 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", , 2020-10-27T04:19:47.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", , 2020-10-27T04:19:47.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", Ignoring malparsed backtrace: 2020-10-27T04:19:47.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.461 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { , 2020-10-27T04:19:47.502 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", , 2020-10-27T04:19:47.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", Ignoring malparsed backtrace: 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], , 2020-10-27T04:19:47.516 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, , 2020-10-27T04:19:47.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", Ignoring malparsed backtrace: 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "diskprediction_port": { , 2020-10-27T04:19:47.544 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.558 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, Ignoring malparsed backtrace: 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", , 2020-10-27T04:19:47.571 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", , 2020-10-27T04:19:47.572 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: { , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", , 2020-10-27T04:19:47.585 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, Ignoring malparsed backtrace: 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { , 2020-10-27T04:19:47.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", Ignoring malparsed backtrace: 2020-10-27T04:19:47.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], Ignoring malparsed backtrace: 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { , 2020-10-27T04:19:47.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", Ignoring malparsed backtrace: 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, , 2020-10-27T04:19:47.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", Ignoring malparsed backtrace: 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { , 2020-10-27T04:19:47.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", Ignoring malparsed backtrace: 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", , 2020-10-27T04:19:47.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] Ignoring malparsed backtrace: 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { , 2020-10-27T04:19:47.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", Ignoring malparsed backtrace: 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] , 2020-10-27T04:19:47.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, Ignoring malparsed backtrace: 2020-10-27T04:19:54.239 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph osd deep-scrub 3 , instructed osd(s) 3 to deep-scrub , 2020-10-27T04:19:54.899 INFO:tasks.ceph:Scrubbing osd.4 Ignoring malparsed backtrace: 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.10 last_scrub_stamp 2020-10-27 04:18:57.342688 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:20:41.270 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:21:22.647 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:21:22.647 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:21:42.649 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:22:24.804 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:22:24.805 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:23:07.878 INFO:tasks.ceph:pgid 1.b last_scrub_stamp 2020-10-27 04:18:54.119312 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=54, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:23:07.879 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:09.996 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:24:09.997 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:24:54.522 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.e , instructing pg 2.es0 on osd.4 to deep-scrub , 2020-10-27T04:24:55.189 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.7 Ignoring malparsed backtrace: 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.c last_scrub_stamp 2020-10-27 04:18:57.140975 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=57, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:25:54.269 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:26:35.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:26:55.674 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json , dumped all Ignoring malparsed backtrace: instructing pg 1.0 on osd.1 to deep-scrub , 2020-10-27T04:27:27.955 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.3 , instructing pg 2.3s0 on osd.3 to deep-scrub Ignoring malparsed backtrace: 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.a last_scrub_stamp 2020-10-27 04:19:03.291964 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=3, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:28:21.608 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:29:23.710 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:29:59.786 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 1.a , instructing pg 1.a on osd.1 to deep-scrub , 2020-10-27T04:30:00.405 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg deep-scrub 2.9 Ignoring malparsed backtrace: 2020-10-27T04:31:08.660 INFO:tasks.ceph:pgid 1.d last_scrub_stamp 2020-10-27 04:18:56.256846 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=56, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:31:08.661 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: dumped all , 2020-10-27T04:32:10.678 INFO:teuthology.orchestra.run.smithi039.stdout: , 2020-10-27T04:32:10.701 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":524,"stamp":"2020-10-27 04:32:09.052368","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":84,"acting":84,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":84,"num_osds":8,"num_per_pool_osds":8,"kb":754974720,"kb_used":8538736,"kb_used_data":150064,"kb_used_omap":7,"kb_used_meta":8388600,"kb_avail":746435984,"statfs":{"total":773094113280,"available":764350447616,"internally_reserved":8589934592,"allocated":153665536,"data_stored":180304914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":8589926562},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003833"},"pg_stats":[{"pgid":"1.11","version":"0'0","reported_seq":"84","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.426833","last_change":"2020-10-27 04:18:55.156939","last_active":"2020-10-27 04:19:34.426833","last_peered":"2020-10-27 04:19:34.426833","last_clean":"2020-10-27 04:19:34.426833","last_became_active":"2020-10-27 04:18:53.912405","last_became_peered":"2020-10-27 04:18:53.912405","last_unstale":"2020-10-27 04:19:34.426833","last_undegraded":"2020-10-27 04:19:34.426833","last_fullsized":"2020-10-27 04:19:34.426833","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.156652","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.156652","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.10","version":"0'0","reported_seq":"38","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602128","last_change":"2020-10-27 04:18:57.342812","last_active":"2020-10-27 04:19:34.602128","last_peered":"2020-10-27 04:19:34.602128","last_clean":"2020-10-27 04:19:34.602128","last_became_active":"2020-10-27 04:18:53.913747","last_became_peered":"2020-10-27 04:18:53.913747","last_unstale":"2020-10-27 04:19:34.602128","last_undegraded":"2020-10-27 04:19:34.602128","last_fullsized":"2020-10-27 04:19:34.602128","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.342688","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.342688","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"0'0","reported_seq":"39","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427250","last_change":"2020-10-27 04:18:57.141096","last_active":"2020-10-27 04:19:34.427250","last_peered":"2020-10-27 04:19:34.427250","last_clean":"2020-10-27 04:19:34.427250","last_became_active":"2020-10-27 04:18:53.913837","last_became_peered":"2020-10-27 04:18:53.913837","last_unstale":"2020-10-27 04:19:34.427250","last_undegraded":"2020-10-27 04:19:34.427250","last_fullsized":"2020-10-27 04:19:34.427250","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:57.140975","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:57.140975","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.f","version":"63'137","reported_seq":"305","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602824","last_change":"2020-10-27 04:18:55.963308","last_active":"2020-10-27 04:19:34.602824","last_peered":"2020-10-27 04:19:34.602824","last_clean":"2020-10-27 04:19:34.602824","last_became_active":"2020-10-27 04:18:14.269110","last_became_peered":"2020-10-27 04:18:14.269110","last_unstale":"2020-10-27 04:19:34.602824","last_undegraded":"2020-10-27 04:19:34.602824","last_fullsized":"2020-10-27 04:19:34.602824","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13731319,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":33,"num_read_kb":5580,"num_write":76,"num_write_kb":18464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.d","version":"0'0","reported_seq":"95","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608614","last_change":"2020-10-27 04:18:56.256991","last_active":"2020-10-27 04:19:34.608614","last_peered":"2020-10-27 04:19:34.608614","last_clean":"2020-10-27 04:19:34.608614","last_became_active":"2020-10-27 04:18:53.917145","last_became_peered":"2020-10-27 04:18:53.917145","last_unstale":"2020-10-27 04:19:34.608614","last_undegraded":"2020-10-27 04:19:34.608614","last_fullsized":"2020-10-27 04:19:34.608614","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:56.256846","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:56.256846","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"55'8","reported_seq":"150","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142999","last_change":"2020-10-27 04:19:27.888673","last_active":"2020-10-27 04:19:34.142999","last_peered":"2020-10-27 04:19:34.142999","last_clean":"2020-10-27 04:19:34.142999","last_became_active":"2020-10-27 04:19:27.887042","last_became_peered":"2020-10-27 04:19:27.887042","last_unstale":"2020-10-27 04:19:34.142999","last_undegraded":"2020-10-27 04:19:34.142999","last_fullsized":"2020-10-27 04:19:34.142999","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":470596,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":1382,"num_write":4,"num_write_kb":460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608889","last_change":"2020-10-27 04:18:58.356254","last_active":"2020-10-27 04:19:34.608889","last_peered":"2020-10-27 04:19:34.608889","last_clean":"2020-10-27 04:19:34.608889","last_became_active":"2020-10-27 04:18:45.994526","last_became_peered":"2020-10-27 04:18:45.994526","last_unstale":"2020-10-27 04:19:34.608889","last_undegraded":"2020-10-27 04:19:34.608889","last_fullsized":"2020-10-27 04:19:34.608889","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:58.356124","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:58.356124","last_clean_scrub_stamp":"2020-10-27 04:18:58.356124","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"65'41","reported_seq":"224","reported_epoch":"88","state":"active+clean+scrubbing","last_fresh":"2020-10-27 04:19:34.605028","last_change":"2020-10-27 04:19:14.461624","last_active":"2020-10-27 04:19:34.605028","last_peered":"2020-10-27 04:19:34.605028","last_clean":"2020-10-27 04:19:34.605028","last_became_active":"2020-10-27 04:18:14.276711","last_became_peered":"2020-10-27 04:18:14.276711","last_unstale":"2020-10-27 04:19:34.605028","last_undegraded":"2020-10-27 04:19:34.605028","last_fullsized":"2020-10-27 04:19:34.605028","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4756381,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":12959,"num_write":40,"num_write_kb":4914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"92","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146936","last_change":"2020-10-27 04:19:14.916785","last_active":"2020-10-27 04:19:34.146936","last_peered":"2020-10-27 04:19:34.146936","last_clean":"2020-10-27 04:19:34.146936","last_became_active":"2020-10-27 04:19:14.915920","last_became_peered":"2020-10-27 04:19:14.915920","last_unstale":"2020-10-27 04:19:34.146936","last_undegraded":"2020-10-27 04:19:34.146936","last_fullsized":"2020-10-27 04:19:34.146936","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:52.955136","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:52.955136","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"73'127","reported_seq":"310","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146900","last_change":"2020-10-27 04:19:17.899883","last_active":"2020-10-27 04:19:34.146900","last_peered":"2020-10-27 04:19:34.146900","last_clean":"2020-10-27 04:19:34.146900","last_became_active":"2020-10-27 04:19:17.882175","last_became_peered":"2020-10-27 04:19:17.882175","last_unstale":"2020-10-27 04:19:34.146900","last_undegraded":"2020-10-27 04:19:34.146900","last_fullsized":"2020-10-27 04:19:34.146900","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":127,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9260657,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":31,"num_read_kb":15822,"num_write":191,"num_write_kb":10827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2016840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"88","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608728","last_change":"2020-10-27 04:18:54.277438","last_active":"2020-10-27 04:19:34.608728","last_peered":"2020-10-27 04:19:34.608728","last_clean":"2020-10-27 04:19:34.608728","last_became_active":"2020-10-27 04:18:45.991230","last_became_peered":"2020-10-27 04:18:45.991230","last_unstale":"2020-10-27 04:19:34.608728","last_undegraded":"2020-10-27 04:19:34.608728","last_fullsized":"2020-10-27 04:19:34.608728","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.277330","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.277330","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"65'30","reported_seq":"222","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.143009","last_change":"2020-10-27 04:19:27.889205","last_active":"2020-10-27 04:19:34.143009","last_peered":"2020-10-27 04:19:34.143009","last_clean":"2020-10-27 04:19:34.143009","last_became_active":"2020-10-27 04:19:27.887303","last_became_peered":"2020-10-27 04:19:27.887303","last_unstale":"2020-10-27 04:19:34.143009","last_undegraded":"2020-10-27 04:19:34.143009","last_fullsized":"2020-10-27 04:19:34.143009","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5159749,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":3999,"num_write":14,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":4789995,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"106","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:54.406100","last_change":"2020-10-27 04:29:54.406213","last_active":"2020-10-27 04:29:54.406100","last_peered":"2020-10-27 04:29:54.406100","last_clean":"2020-10-27 04:29:54.406100","last_became_active":"2020-10-27 04:18:45.995583","last_became_peered":"2020-10-27 04:18:45.995583","last_unstale":"2020-10-27 04:29:54.406100","last_undegraded":"2020-10-27 04:29:54.406100","last_fullsized":"2020-10-27 04:29:54.406100","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:54.406082","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:54.406082","last_clean_scrub_stamp":"2020-10-27 04:29:54.406082","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"55'50","reported_seq":"228","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147332","last_change":"2020-10-27 04:19:14.929768","last_active":"2020-10-27 04:19:34.147332","last_peered":"2020-10-27 04:19:34.147332","last_clean":"2020-10-27 04:19:34.147332","last_became_active":"2020-10-27 04:19:14.926647","last_became_peered":"2020-10-27 04:19:14.926647","last_unstale":"2020-10-27 04:19:34.147332","last_undegraded":"2020-10-27 04:19:34.147332","last_fullsized":"2020-10-27 04:19:34.147332","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2302559,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":4243,"num_write":22,"num_write_kb":7245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"87","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608346","last_change":"2020-10-27 04:18:55.292214","last_active":"2020-10-27 04:19:34.608346","last_peered":"2020-10-27 04:19:34.608346","last_clean":"2020-10-27 04:19:34.608346","last_became_active":"2020-10-27 04:18:45.992519","last_became_peered":"2020-10-27 04:18:45.992519","last_unstale":"2020-10-27 04:19:34.608346","last_undegraded":"2020-10-27 04:19:34.608346","last_fullsized":"2020-10-27 04:19:34.608346","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:55.292162","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:55.292162","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"55'64","reported_seq":"236","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146895","last_change":"2020-10-27 04:19:14.925901","last_active":"2020-10-27 04:19:34.146895","last_peered":"2020-10-27 04:19:34.146895","last_clean":"2020-10-27 04:19:34.146895","last_became_active":"2020-10-27 04:19:14.922144","last_became_peered":"2020-10-27 04:19:14.922144","last_unstale":"2020-10-27 04:19:34.146895","last_undegraded":"2020-10-27 04:19:34.146895","last_fullsized":"2020-10-27 04:19:34.146895","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":64,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6100941,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":68,"num_write_kb":6525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"82","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146863","last_change":"2020-10-27 04:19:14.916148","last_active":"2020-10-27 04:19:34.146863","last_peered":"2020-10-27 04:19:34.146863","last_clean":"2020-10-27 04:19:34.146863","last_became_active":"2020-10-27 04:19:14.914681","last_became_peered":"2020-10-27 04:19:14.914681","last_unstale":"2020-10-27 04:19:34.146863","last_undegraded":"2020-10-27 04:19:34.146863","last_fullsized":"2020-10-27 04:19:34.146863","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:17:46.831755","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:17:46.831755","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'110","reported_seq":"284","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146939","last_change":"2020-10-27 04:19:17.889774","last_active":"2020-10-27 04:19:34.146939","last_peered":"2020-10-27 04:19:34.146939","last_clean":"2020-10-27 04:19:34.146939","last_became_active":"2020-10-27 04:19:17.885352","last_became_peered":"2020-10-27 04:19:17.885352","last_unstale":"2020-10-27 04:19:34.146939","last_undegraded":"2020-10-27 04:19:34.146939","last_fullsized":"2020-10-27 04:19:34.146939","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":110,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11563216,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":1195,"num_write":102,"num_write_kb":11958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"64'43","reported_seq":"227","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.140212","last_change":"2020-10-27 04:18:55.937872","last_active":"2020-10-27 04:19:34.140212","last_peered":"2020-10-27 04:19:34.140212","last_clean":"2020-10-27 04:19:34.140212","last_became_active":"2020-10-27 04:18:14.271076","last_became_peered":"2020-10-27 04:18:14.271076","last_unstale":"2020-10-27 04:19:34.140212","last_undegraded":"2020-10-27 04:19:34.140212","last_fullsized":"2020-10-27 04:19:34.140212","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":43,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6050520,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":8972,"num_write":17,"num_write_kb":7783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"104","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:59.341431","last_change":"2020-10-27 04:29:59.341513","last_active":"2020-10-27 04:29:59.341431","last_peered":"2020-10-27 04:29:59.341431","last_clean":"2020-10-27 04:29:59.341431","last_became_active":"2020-10-27 04:18:45.997645","last_became_peered":"2020-10-27 04:18:45.997645","last_unstale":"2020-10-27 04:29:59.341431","last_undegraded":"2020-10-27 04:29:59.341431","last_fullsized":"2020-10-27 04:29:59.341431","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:59.341421","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:59.341421","last_clean_scrub_stamp":"2020-10-27 04:29:59.341421","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"40'11","reported_seq":"182","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146908","last_change":"2020-10-27 04:19:14.927715","last_active":"2020-10-27 04:19:34.146908","last_peered":"2020-10-27 04:19:34.146908","last_clean":"2020-10-27 04:19:34.146908","last_became_active":"2020-10-27 04:19:14.921996","last_became_peered":"2020-10-27 04:19:14.921996","last_unstale":"2020-10-27 04:19:34.146908","last_undegraded":"2020-10-27 04:19:34.146908","last_fullsized":"2020-10-27 04:19:34.146908","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"89","reported_epoch":"89","state":"active+clean","last_fresh":"2020-10-27 04:29:58.601607","last_change":"2020-10-27 04:29:58.601770","last_active":"2020-10-27 04:29:58.601607","last_peered":"2020-10-27 04:29:58.601607","last_clean":"2020-10-27 04:29:58.601607","last_became_active":"2020-10-27 04:19:27.881858","last_became_peered":"2020-10-27 04:19:27.881858","last_unstale":"2020-10-27 04:29:58.601607","last_undegraded":"2020-10-27 04:29:58.601607","last_fullsized":"2020-10-27 04:29:58.601607","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:29:58.601579","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:29:58.601579","last_clean_scrub_stamp":"2020-10-27 04:29:58.601579","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"0'0","reported_seq":"73","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142619","last_change":"2020-10-27 04:19:27.888587","last_active":"2020-10-27 04:19:34.142619","last_peered":"2020-10-27 04:19:34.142619","last_clean":"2020-10-27 04:19:34.142619","last_became_active":"2020-10-27 04:19:27.887602","last_became_peered":"2020-10-27 04:19:27.887602","last_unstale":"2020-10-27 04:19:34.142619","last_undegraded":"2020-10-27 04:19:34.142619","last_fullsized":"2020-10-27 04:19:34.142619","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:54.119312","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:18:54.119312","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"63'111","reported_seq":"286","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.602263","last_change":"2020-10-27 04:18:55.939797","last_active":"2020-10-27 04:19:34.602263","last_peered":"2020-10-27 04:19:34.602263","last_clean":"2020-10-27 04:19:34.602263","last_became_active":"2020-10-27 04:18:14.266566","last_became_peered":"2020-10-27 04:18:14.266566","last_unstale":"2020-10-27 04:19:34.602263","last_undegraded":"2020-10-27 04:19:34.602263","last_fullsized":"2020-10-27 04:19:34.602263","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14773353,"num_objects":7,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":31,"num_read_kb":2334,"num_write":144,"num_write_kb":16198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608318","last_change":"2020-10-27 04:19:03.292079","last_active":"2020-10-27 04:19:34.608318","last_peered":"2020-10-27 04:19:34.608318","last_clean":"2020-10-27 04:19:34.608318","last_became_active":"2020-10-27 04:18:53.919537","last_became_peered":"2020-10-27 04:18:53.919537","last_unstale":"2020-10-27 04:19:34.608318","last_undegraded":"2020-10-27 04:19:34.608318","last_fullsized":"2020-10-27 04:19:34.608318","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:03.291964","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:03.291964","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"55'28","reported_seq":"175","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.142751","last_change":"2020-10-27 04:19:27.888943","last_active":"2020-10-27 04:19:34.142751","last_peered":"2020-10-27 04:19:34.142751","last_clean":"2020-10-27 04:19:34.142751","last_became_active":"2020-10-27 04:19:27.887163","last_became_peered":"2020-10-27 04:19:27.887163","last_unstale":"2020-10-27 04:19:34.142751","last_undegraded":"2020-10-27 04:19:34.142751","last_fullsized":"2020-10-27 04:19:34.142751","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4218557,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":1091,"num_write":16,"num_write_kb":4354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608526","last_change":"2020-10-27 04:19:13.189378","last_active":"2020-10-27 04:19:34.608526","last_peered":"2020-10-27 04:19:34.608526","last_clean":"2020-10-27 04:19:34.608526","last_became_active":"2020-10-27 04:18:53.917040","last_became_peered":"2020-10-27 04:18:53.917040","last_unstale":"2020-10-27 04:19:34.608526","last_undegraded":"2020-10-27 04:19:34.608526","last_fullsized":"2020-10-27 04:19:34.608526","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:13.189245","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:13.189245","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"50'22","reported_seq":"195","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427169","last_change":"2020-10-27 04:18:55.910655","last_active":"2020-10-27 04:19:34.427169","last_peered":"2020-10-27 04:19:34.427169","last_clean":"2020-10-27 04:19:34.427169","last_became_active":"2020-10-27 04:18:14.278718","last_became_peered":"2020-10-27 04:18:14.278718","last_unstale":"2020-10-27 04:19:34.427169","last_undegraded":"2020-10-27 04:19:34.427169","last_fullsized":"2020-10-27 04:19:34.427169","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":235298,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":44,"num_write_kb":1084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"94","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608286","last_change":"2020-10-27 04:19:01.325786","last_active":"2020-10-27 04:19:34.608286","last_peered":"2020-10-27 04:19:34.608286","last_clean":"2020-10-27 04:19:34.608286","last_became_active":"2020-10-27 04:18:53.915676","last_became_peered":"2020-10-27 04:18:53.915676","last_unstale":"2020-10-27 04:19:34.608286","last_undegraded":"2020-10-27 04:19:34.608286","last_fullsized":"2020-10-27 04:19:34.608286","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:01.325624","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:01.325624","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"64'56","reported_seq":"226","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.146963","last_change":"2020-10-27 04:19:17.888785","last_active":"2020-10-27 04:19:34.146963","last_peered":"2020-10-27 04:19:34.146963","last_clean":"2020-10-27 04:19:34.146963","last_became_active":"2020-10-27 04:19:17.885684","last_became_peered":"2020-10-27 04:19:17.885684","last_unstale":"2020-10-27 04:19:34.146963","last_undegraded":"2020-10-27 04:19:34.146963","last_fullsized":"2020-10-27 04:19:34.146963","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":56,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":2026,"num_write":41,"num_write_kb":5997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"0'0","reported_seq":"41","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.147288","last_change":"2020-10-27 04:19:17.548013","last_active":"2020-10-27 04:19:34.147288","last_peered":"2020-10-27 04:19:34.147288","last_clean":"2020-10-27 04:19:34.147288","last_became_active":"2020-10-27 04:19:14.920337","last_became_peered":"2020-10-27 04:19:14.920337","last_unstale":"2020-10-27 04:19:34.147288","last_undegraded":"2020-10-27 04:19:34.147288","last_fullsized":"2020-10-27 04:19:34.147288","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:17.547918","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:17:46.831755","last_clean_scrub_stamp":"2020-10-27 04:19:17.547918","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"50'26","reported_seq":"202","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.427394","last_change":"2020-10-27 04:18:55.909529","last_active":"2020-10-27 04:19:34.427394","last_peered":"2020-10-27 04:19:34.427394","last_clean":"2020-10-27 04:19:34.427394","last_became_active":"2020-10-27 04:18:14.273127","last_became_peered":"2020-10-27 04:18:14.273127","last_unstale":"2020-10-27 04:19:34.427394","last_undegraded":"2020-10-27 04:19:34.427394","last_fullsized":"2020-10-27 04:19:34.427394","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":547,"num_write":10,"num_write_kb":6896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"0'0","reported_seq":"37","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.608636","last_change":"2020-10-27 04:19:07.322883","last_active":"2020-10-27 04:19:34.608636","last_peered":"2020-10-27 04:19:34.608636","last_clean":"2020-10-27 04:19:34.608636","last_became_active":"2020-10-27 04:18:53.920956","last_became_peered":"2020-10-27 04:18:53.920956","last_unstale":"2020-10-27 04:19:34.608636","last_undegraded":"2020-10-27 04:19:34.608636","last_fullsized":"2020-10-27 04:19:34.608636","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:19:07.322732","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:19:07.322732","last_clean_scrub_stamp":"2020-10-27 04:19:07.322732","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.d","version":"55'62","reported_seq":"242","reported_epoch":"88","state":"active+clean","last_fresh":"2020-10-27 04:19:34.604887","last_change":"2020-10-27 04:18:55.917852","last_active":"2020-10-27 04:19:34.604887","last_peered":"2020-10-27 04:19:34.604887","last_clean":"2020-10-27 04:19:34.604887","last_became_active":"2020-10-27 04:18:14.266855","last_became_peered":"2020-10-27 04:18:14.266855","last_unstale":"2020-10-27 04:19:34.604887","last_undegraded":"2020-10-27 04:19:34.604887","last_fullsized":"2020-10-27 04:19:34.604887","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-10-27 04:18:13.242061","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-10-27 04:18:13.242061","last_clean_scrub_stamp":"2020-10-27 04:18:13.242061","log_size":62,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2201717,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":7644,"num_write":22,"num_write_kb":3533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":93144394,"num_objects":62,"num_object_clones":18,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":2,"num_read":287,"num_read_kb":67803,"num_write":814,"num_write_kb":113350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":6806835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145244160,"data_stored":176377794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":926,"ondisk_log_size":926,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":18,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":36,"acting":36,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575026,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1060788,"kb_used_data":12204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93311052,"statfs":{"total":96636764160,"available":95550517248,"internally_reserved":1073741824,"allocated":12496896,"data_stored":12292194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:15 2020","interfaces":[{"interface":"back","average":{"1min":49.296999999999997,"5min":12.644,"15min":5.9509999999999996},"min":{"1min":1.7649999999999999,"5min":1.2589999999999999,"15min":1.218},"max":{"1min":985.40999999999997,"5min":985.40999999999997,"15min":985.40999999999997},"last":2.218},{"interface":"front","average":{"1min":2.452,"5min":2.4180000000000001,"15min":10.823},"min":{"1min":1.369,"5min":1.2030000000000001,"15min":1.1140000000000001},"max":{"1min":3.79,"5min":3.9300000000000002,"15min":971.95100000000002},"last":2.8319999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.0529999999999999,"5min":11.887,"15min":8.0559999999999992},"min":{"1min":1.196,"5min":1.196,"15min":1.139},"max":{"1min":3.0030000000000001,"5min":834.52300000000002,"15min":834.52300000000002},"last":1.782},{"interface":"front","average":{"1min":2.6219999999999999,"5min":10.395,"15min":7.609},"min":{"1min":2.1320000000000001,"5min":1.589,"15min":1.095},"max":{"1min":3.218,"5min":621.11199999999997,"15min":675.13099999999997},"last":2.2949999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":37.174999999999997,"5min":9.423,"15min":10.127000000000001},"min":{"1min":1.8260000000000001,"5min":1.5900000000000001,"15min":1.2649999999999999},"max":{"1min":595.22699999999998,"5min":595.22699999999998,"15min":749.68899999999996},"last":2.673},{"interface":"front","average":{"1min":26.971,"5min":7.2380000000000004,"15min":7.1200000000000001},"min":{"1min":1.7769999999999999,"5min":1.5860000000000001,"15min":1.1799999999999999},"max":{"1min":423.61200000000002,"5min":423.61200000000002,"15min":503.75799999999998},"last":1.861}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.7709999999999999,"5min":10.558,"15min":5.4829999999999997},"min":{"1min":2.0529999999999999,"5min":1.9239999999999999,"15min":1.6120000000000001},"max":{"1min":3.29,"5min":810.279,"15min":810.279},"last":2.262},{"interface":"front","average":{"1min":2.5219999999999998,"5min":2.6219999999999999,"15min":19.773},"min":{"1min":1.694,"5min":1.6879999999999999,"15min":1.552},"max":{"1min":3.1059999999999999,"5min":3.859,"15min":897.61800000000005},"last":2.5640000000000001}]},{"osd":4,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8399999999999999,"5min":22.492999999999999,"15min":13.928000000000001},"min":{"1min":2.3260000000000001,"5min":1.845,"15min":1.5569999999999999},"max":{"1min":3.2559999999999998,"5min":671.45899999999995,"15min":671.45899999999995},"last":2.887},{"interface":"front","average":{"1min":2.706,"5min":11.789999999999999,"15min":9.1460000000000008},"min":{"1min":1.931,"5min":1.931,"15min":1.665},"max":{"1min":3.2000000000000002,"5min":639.77499999999998,"15min":639.77499999999998},"last":2.758}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.7120000000000002,"5min":4.444,"15min":5.7809999999999997},"min":{"1min":1.8460000000000001,"5min":1.7,"15min":1.7},"max":{"1min":3.2709999999999999,"5min":159.59399999999999,"15min":735.70500000000004},"last":3.0699999999999998},{"interface":"front","average":{"1min":3.0550000000000002,"5min":16.914999999999999,"15min":16.773},"min":{"1min":2.343,"5min":2.343,"15min":2.1909999999999998},"max":{"1min":3.6560000000000001,"5min":923.20100000000002,"15min":923.20100000000002},"last":2.343}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.3700000000000001,"5min":15.603,"15min":10.483000000000001},"min":{"1min":2.581,"5min":2.581,"15min":2.2679999999999998},"max":{"1min":5.2469999999999999,"5min":992.976,"15min":992.976},"last":679.41899999999998},{"interface":"front","average":{"1min":3.0049999999999999,"5min":6.5810000000000004,"15min":4.5540000000000003},"min":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":1.9530000000000001},"max":{"1min":3.8420000000000001,"5min":390.29300000000001,"15min":390.29300000000001},"last":3.1309999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575025,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1067020,"kb_used_data":18436,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93304820,"statfs":{"total":96636764160,"available":95544135680,"internally_reserved":1073741824,"allocated":18878464,"data_stored":23929015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.2120000000000002,"5min":2.387,"15min":2.395},"min":{"1min":1.641,"5min":1.571,"15min":1.022},"max":{"1min":3.0219999999999998,"5min":3.4750000000000001,"15min":4.4809999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":2.3140000000000001,"5min":2.5310000000000001,"15min":2.5409999999999999},"min":{"1min":1.53,"5min":1.53,"15min":1.083},"max":{"1min":3.5670000000000002,"5min":3.794,"15min":7.4249999999999998},"last":3.0099999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.109,"15min":2.1930000000000001},"min":{"1min":1.198,"5min":1.175,"15min":1.115},"max":{"1min":2.7810000000000001,"5min":3.2370000000000001,"15min":10.25},"last":2.8330000000000002},{"interface":"front","average":{"1min":2.7709999999999999,"5min":15.755000000000001,"15min":6.71},"min":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.0549999999999999},"max":{"1min":3.649,"5min":1340.4169999999999,"15min":1340.4169999999999},"last":1.724}]},{"osd":2,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.5190000000000001,"5min":2.597,"15min":9.9090000000000007},"min":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.452},"max":{"1min":3.1579999999999999,"5min":3.621,"15min":900.75599999999997},"last":2.367},{"interface":"front","average":{"1min":3.2090000000000001,"5min":2.8330000000000002,"15min":2.7799999999999998},"min":{"1min":2.3420000000000001,"5min":1.605,"15min":1.452},"max":{"1min":4.133,"5min":4.133,"15min":4.7000000000000002},"last":2.7690000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:40 2020","interfaces":[{"interface":"back","average":{"1min":2.6970000000000001,"5min":2.8479999999999999,"15min":7.6399999999999997},"min":{"1min":2.0139999999999998,"5min":1.7370000000000001,"15min":1.607},"max":{"1min":3.6840000000000002,"5min":4.452,"15min":637.64599999999996},"last":3.407},{"interface":"front","average":{"1min":3.0979999999999999,"5min":12.949999999999999,"15min":8.9559999999999995},"min":{"1min":2.2290000000000001,"5min":1.7989999999999999,"15min":1.7989999999999999},"max":{"1min":3.8940000000000001,"5min":1002.468,"15min":1002.468},"last":2.895}]},{"osd":4,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.4630000000000001,"5min":2.589,"15min":10.563000000000001},"min":{"1min":1.6459999999999999,"5min":1.462,"15min":1.452},"max":{"1min":3.0089999999999999,"5min":3.7280000000000002,"15min":888.96900000000005},"last":1.79},{"interface":"front","average":{"1min":2.835,"5min":9.891,"15min":7.2279999999999998},"min":{"1min":1.498,"5min":1.498,"15min":1.498},"max":{"1min":3.802,"5min":700.48099999999999,"15min":700.48099999999999},"last":3.1930000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":2.9039999999999999,"5min":2.9630000000000001,"15min":5.8529999999999998},"min":{"1min":2.2519999999999998,"5min":1.506,"15min":1.506},"max":{"1min":3.7200000000000002,"5min":4.3019999999999996,"15min":552.64099999999996},"last":3.1160000000000001},{"interface":"front","average":{"1min":3.3159999999999998,"5min":12.209,"15min":6.0279999999999996},"min":{"1min":2.3119999999999998,"5min":1.99,"15min":1.488},"max":{"1min":4.0350000000000001,"5min":912.10799999999995,"15min":912.10799999999995},"last":3.355}]},{"osd":7,"last update":"Tue Oct 27 04:31:58 2020","interfaces":[{"interface":"back","average":{"1min":3.0499999999999998,"5min":23.687000000000001,"15min":11.260999999999999},"min":{"1min":2.1480000000000001,"5min":2.1480000000000001,"15min":1.52},"max":{"1min":3.786,"5min":787.13199999999995,"15min":787.13199999999995},"last":3.2789999999999999},{"interface":"front","average":{"1min":3.569,"5min":3.2679999999999998,"15min":9.1240000000000006},"min":{"1min":2.871,"5min":2.492,"15min":1.966},"max":{"1min":4.6680000000000001,"5min":4.6680000000000001,"15min":1000.52},"last":2.6899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575025,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1068864,"kb_used_data":20280,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93302976,"statfs":{"total":96636764160,"available":95542247424,"internally_reserved":1073741824,"allocated":20766720,"data_stored":26346047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.6120000000000001,"5min":2.4689999999999999,"15min":6.6299999999999999},"min":{"1min":1.3720000000000001,"5min":1.302,"15min":1.0449999999999999},"max":{"1min":3.3140000000000001,"5min":3.8340000000000001,"15min":787.62400000000002},"last":2.7050000000000001},{"interface":"front","average":{"1min":2.2050000000000001,"5min":12.976000000000001,"15min":14.279999999999999},"min":{"1min":1.4490000000000001,"5min":1.226,"15min":0.89600000000000002},"max":{"1min":3.1429999999999998,"5min":910.65099999999995,"15min":910.65099999999995},"last":2.4209999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.3570000000000002,"5min":2.5209999999999999,"15min":4.1239999999999997},"min":{"1min":1.591,"5min":1.474,"15min":1.194},"max":{"1min":3.411,"5min":3.8730000000000002,"15min":484.69600000000003},"last":785.25099999999998},{"interface":"front","average":{"1min":2.387,"5min":11.192,"15min":18.960000000000001},"min":{"1min":1.522,"5min":1.522,"15min":1.3979999999999999},"max":{"1min":2.887,"5min":834.95000000000005,"15min":877.15300000000002},"last":3.8279999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":19.675999999999998,"5min":6.1299999999999999,"15min":3.7130000000000001},"min":{"1min":1.972,"5min":1.52,"15min":1.2509999999999999},"max":{"1min":323.01799999999997,"5min":323.01799999999997,"15min":323.01799999999997},"last":2.7610000000000001},{"interface":"front","average":{"1min":2.5329999999999999,"5min":8.093,"15min":6.774},"min":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.238},"max":{"1min":4.3150000000000004,"5min":599.65599999999995,"15min":599.65599999999995},"last":3.9020000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:37 2020","interfaces":[{"interface":"back","average":{"1min":2.996,"5min":2.8809999999999998,"15min":2.996},"min":{"1min":1.6950000000000001,"5min":1.661,"15min":1.661},"max":{"1min":3.8069999999999999,"5min":3.899,"15min":4.6299999999999999},"last":2.798},{"interface":"front","average":{"1min":3.0310000000000001,"5min":4.9000000000000004,"15min":4.4480000000000004},"min":{"1min":2.0419999999999998,"5min":1.7190000000000001,"15min":1.3220000000000001},"max":{"1min":4.4039999999999999,"5min":218.10400000000001,"15min":228.80099999999999},"last":3.9889999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.1080000000000001,"5min":3.0630000000000002,"15min":2.968},"min":{"1min":2.399,"5min":2.2349999999999999,"15min":1.5149999999999999},"max":{"1min":3.9729999999999999,"5min":4.1669999999999998,"15min":4.633},"last":3.0600000000000001},{"interface":"front","average":{"1min":2.895,"5min":2.8479999999999999,"15min":5.8499999999999996},"min":{"1min":1.8220000000000001,"5min":1.655,"15min":1.615},"max":{"1min":4.3680000000000003,"5min":4.3680000000000003,"15min":623.83100000000002},"last":4.0570000000000004}]},{"osd":6,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":3.2050000000000001,"5min":9.7910000000000004,"15min":10.93},"min":{"1min":2.4510000000000001,"5min":2.2799999999999998,"15min":1.7110000000000001},"max":{"1min":3.895,"5min":727.97400000000005,"15min":816.51300000000003},"last":2.835},{"interface":"front","average":{"1min":2.8940000000000001,"5min":18.178999999999998,"15min":12.436},"min":{"1min":2.2229999999999999,"5min":1.8320000000000001,"15min":1.5700000000000001},"max":{"1min":3.359,"5min":994.68299999999999,"15min":994.68299999999999},"last":2.4940000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:31:08 2020","interfaces":[{"interface":"back","average":{"1min":2.8940000000000001,"5min":6.3639999999999999,"15min":8.2070000000000007},"min":{"1min":2.3940000000000001,"5min":2.0190000000000001,"15min":1.6919999999999999},"max":{"1min":3.496,"5min":346.387,"15min":504.964},"last":2.871},{"interface":"front","average":{"1min":2.988,"5min":3.081,"15min":13.677},"min":{"1min":2.121,"5min":1.9299999999999999,"15min":1.2769999999999999},"max":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":891.01599999999996},"last":4.125}]}]},{"osd":4,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064972,"kb_used_data":16388,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306868,"statfs":{"total":96636764160,"available":95546232832,"internally_reserved":1073741824,"allocated":16781312,"data_stored":20017524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":14.038,"5min":4.6779999999999999,"15min":3.419},"min":{"1min":1.141,"5min":1.141,"15min":1.054},"max":{"1min":261.02499999999998,"5min":261.02499999999998,"15min":261.02499999999998},"last":1.633},{"interface":"front","average":{"1min":2.7450000000000001,"5min":2.512,"15min":17.265999999999998},"min":{"1min":1.3899999999999999,"5min":1.234,"15min":1.1419999999999999},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":808.98800000000006},"last":2.1509999999999998}]},{"osd":1,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":6.6239999999999997,"15min":6.2779999999999996},"min":{"1min":1.702,"5min":1.4510000000000001,"15min":1.0980000000000001},"max":{"1min":4.2809999999999997,"5min":454.637,"15min":654.36699999999996},"last":3.218},{"interface":"front","average":{"1min":3.145,"5min":2.6920000000000002,"15min":5.0830000000000002},"min":{"1min":1.8420000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"max":{"1min":5.4420000000000002,"5min":5.4420000000000002,"15min":949.18700000000001},"last":2.2549999999999999}]},{"osd":2,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.351,"5min":7.1449999999999996,"15min":12.432},"min":{"1min":1.4350000000000001,"5min":1.4219999999999999,"15min":1.165},"max":{"1min":5.0810000000000004,"5min":451.41800000000001,"15min":976.43700000000001},"last":2.956},{"interface":"front","average":{"1min":2.6200000000000001,"5min":2.6419999999999999,"15min":11.617000000000001},"min":{"1min":1.754,"5min":1.754,"15min":1.623},"max":{"1min":3.9820000000000002,"5min":3.9820000000000002,"15min":780.63900000000001},"last":3.3769999999999998}]},{"osd":3,"last update":"Tue Oct 27 04:31:33 2020","interfaces":[{"interface":"back","average":{"1min":2.8029999999999999,"5min":2.6269999999999998,"15min":5.4489999999999998},"min":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.3759999999999999},"max":{"1min":3.6629999999999998,"5min":4.8710000000000004,"15min":478.51299999999998},"last":3.5529999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":6.0709999999999997,"15min":3.8820000000000001},"min":{"1min":2.2709999999999999,"5min":1.75,"15min":1.593},"max":{"1min":4.3310000000000004,"5min":344.82600000000002,"15min":344.82600000000002},"last":3.6400000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.867,"5min":13.503,"15min":7.8840000000000003},"min":{"1min":1.9970000000000001,"5min":1.9730000000000001,"15min":1.3819999999999999},"max":{"1min":4.0209999999999999,"5min":800.56899999999996,"15min":800.56899999999996},"last":3.3140000000000001},{"interface":"front","average":{"1min":19.757999999999999,"5min":14.157,"15min":7.9299999999999997},"min":{"1min":2.2829999999999999,"5min":1.8340000000000001,"15min":1.8340000000000001},"max":{"1min":286.78800000000001,"5min":795.65599999999995,"15min":795.65599999999995},"last":3.5939999999999999}]},{"osd":6,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":2.8820000000000001,"5min":2.7810000000000001,"15min":7.1779999999999999},"min":{"1min":2.0089999999999999,"5min":1.583,"15min":1.583},"max":{"1min":5.2489999999999997,"5min":5.8540000000000001,"15min":872.47000000000003},"last":3.036},{"interface":"front","average":{"1min":3.0329999999999999,"5min":3.0579999999999998,"15min":3.0230000000000001},"min":{"1min":2.089,"5min":2.089,"15min":1.298},"max":{"1min":3.851,"5min":5.2930000000000001,"15min":5.2930000000000001},"last":3.4820000000000002}]},{"osd":7,"last update":"Tue Oct 27 04:32:04 2020","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":6.2850000000000001,"15min":7.04},"min":{"1min":2.194,"5min":1.9630000000000001,"15min":1.9630000000000001},"max":{"1min":5.2960000000000003,"5min":323.93400000000003,"15min":882.08600000000001},"last":3.4300000000000002},{"interface":"front","average":{"1min":3.625,"5min":3.5209999999999999,"15min":6.3170000000000002},"min":{"1min":2.883,"5min":2.2669999999999999,"15min":1.8460000000000001},"max":{"1min":4.3700000000000001,"5min":26.504000000000001,"15min":926.90700000000004},"last":3.6760000000000002}]}]},{"osd":0,"up_from":13,"seq":55834575025,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073392,"kb_used_data":24808,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93298448,"statfs":{"total":96636764160,"available":95537610752,"internally_reserved":1073741824,"allocated":25403392,"data_stored":30154857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":7.9109999999999996,"15min":4.1260000000000003},"min":{"1min":1.466,"5min":1.1080000000000001,"15min":0.90600000000000003},"max":{"1min":3.4100000000000001,"5min":545.69000000000005,"15min":545.69000000000005},"last":1.8220000000000001},{"interface":"front","average":{"1min":2.7799999999999998,"5min":16.379000000000001,"15min":7.0330000000000004},"min":{"1min":1.6259999999999999,"5min":1.145,"15min":0.80900000000000005},"max":{"1min":3.7789999999999999,"5min":658.68899999999996,"15min":658.68899999999996},"last":2.7639999999999998}]},{"osd":2,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.3730000000000002,"5min":4.7110000000000003,"15min":2.8610000000000002},"min":{"1min":1.3440000000000001,"5min":1.212,"15min":0.76900000000000002},"max":{"1min":3.4079999999999999,"5min":272.61700000000002,"15min":272.61700000000002},"last":2.2610000000000001},{"interface":"front","average":{"1min":2.7120000000000002,"5min":6.8280000000000003,"15min":5.0869999999999997},"min":{"1min":1.893,"5min":1.141,"15min":0.97099999999999997},"max":{"1min":3.718,"5min":251.756,"15min":251.756},"last":2.879}]},{"osd":3,"last update":"Tue Oct 27 04:31:38 2020","interfaces":[{"interface":"back","average":{"1min":60.173000000000002,"5min":13.944000000000001,"15min":13.381},"min":{"1min":1.9079999999999999,"5min":1.365,"15min":1.2030000000000001},"max":{"1min":866.04100000000005,"5min":866.04100000000005,"15min":923.22299999999996},"last":1.7829999999999999},{"interface":"front","average":{"1min":2.7330000000000001,"5min":14.502000000000001,"15min":8.968},"min":{"1min":1.8740000000000001,"5min":1.4970000000000001,"15min":1.26},"max":{"1min":3.5369999999999999,"5min":796.69100000000003,"15min":796.69100000000003},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":42.18,"5min":10.504,"15min":13.028},"min":{"1min":2.0569999999999999,"5min":1.167,"15min":1.167},"max":{"1min":712.15599999999995,"5min":712.15599999999995,"15min":809.928},"last":2.8279999999999998},{"interface":"front","average":{"1min":2.6699999999999999,"5min":2.5510000000000002,"15min":6.2770000000000001},"min":{"1min":1.8460000000000001,"5min":1.401,"15min":1.1379999999999999},"max":{"1min":4.9550000000000001,"5min":4.9550000000000001,"15min":636.61699999999996},"last":2.9830000000000001}]},{"osd":5,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.6110000000000002,"5min":2.593,"15min":6.6749999999999998},"min":{"1min":1.9430000000000001,"5min":1.667,"15min":1.4079999999999999},"max":{"1min":3.7309999999999999,"5min":4.9260000000000002,"15min":577.61300000000006},"last":2.9350000000000001},{"interface":"front","average":{"1min":56.189,"5min":21.494,"15min":9.8610000000000007},"min":{"1min":1.946,"5min":1.5449999999999999,"15min":1.5449999999999999},"max":{"1min":961.779,"5min":961.779,"15min":961.779},"last":3.0800000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.8999999999999999,"5min":9.3569999999999993,"15min":12.731999999999999},"min":{"1min":1.988,"5min":1.8859999999999999,"15min":1.276},"max":{"1min":3.512,"5min":503.51400000000001,"15min":824.16200000000003},"last":3.4289999999999998},{"interface":"front","average":{"1min":3.1749999999999998,"5min":3.1589999999999998,"15min":6.5590000000000002},"min":{"1min":2.4260000000000002,"5min":1.702,"15min":1.5600000000000001},"max":{"1min":3.8519999999999999,"5min":4.6280000000000001,"15min":906.19299999999998},"last":3.871}]},{"osd":7,"last update":"Tue Oct 27 04:31:09 2020","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":2.8140000000000001,"15min":3.0979999999999999},"min":{"1min":2.222,"5min":1.73,"15min":1.5760000000000001},"max":{"1min":3.77,"5min":4.1669999999999998,"15min":12.324},"last":3.3690000000000002},{"interface":"front","average":{"1min":33.110999999999997,"5min":9.2379999999999995,"15min":6.2720000000000002},"min":{"1min":2.6469999999999998,"5min":1.9750000000000001,"15min":1.4390000000000001},"max":{"1min":510.59899999999999,"5min":510.59899999999999,"15min":510.59899999999999},"last":3.9260000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1065520,"kb_used_data":16936,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306320,"statfs":{"total":96636764160,"available":95545671680,"internally_reserved":1073741824,"allocated":17342464,"data_stored":24243891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":1.9950000000000001,"5min":1.8640000000000001,"15min":2.7629999999999999},"min":{"1min":1.603,"5min":1.226,"15min":0.72199999999999998},"max":{"1min":2.8530000000000002,"5min":2.8530000000000002,"15min":194.143},"last":1.677},{"interface":"front","average":{"1min":2.6930000000000001,"5min":11.317,"15min":10.750999999999999},"min":{"1min":1.9590000000000001,"5min":1.353,"15min":0.81299999999999994},"max":{"1min":4.0609999999999999,"5min":656.79600000000005,"15min":731.65800000000002},"last":2.3330000000000002}]},{"osd":2,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.0459999999999998,"5min":12.202,"15min":12.997},"min":{"1min":1.5920000000000001,"5min":1.3089999999999999,"15min":0.70199999999999996},"max":{"1min":2.895,"5min":992.125,"15min":992.125},"last":1.7569999999999999},{"interface":"front","average":{"1min":2.7709999999999999,"5min":9.4710000000000001,"15min":5.9850000000000003},"min":{"1min":1.52,"5min":1.3640000000000001,"15min":0.68500000000000005},"max":{"1min":3.6920000000000002,"5min":727.048,"15min":727.048},"last":2.5110000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:36 2020","interfaces":[{"interface":"back","average":{"1min":5.1799999999999997,"5min":3.4060000000000001,"15min":18.274000000000001},"min":{"1min":1.4119999999999999,"5min":1.3720000000000001,"15min":0.92400000000000004},"max":{"1min":61.938000000000002,"5min":63.479999999999997,"15min":879.947},"last":2.9489999999999998},{"interface":"front","average":{"1min":2.6440000000000001,"5min":9.3989999999999991,"15min":8.5289999999999999},"min":{"1min":1.6319999999999999,"5min":1.595,"15min":1.1379999999999999},"max":{"1min":3.4289999999999998,"5min":466.53800000000001,"15min":466.53800000000001},"last":2.2589999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.5659999999999998,"5min":2.4849999999999999,"15min":3.8580000000000001},"min":{"1min":1.6579999999999999,"5min":1.464,"15min":1.224},"max":{"1min":3.7919999999999998,"5min":3.7919999999999998,"15min":325.25},"last":2.5790000000000002},{"interface":"front","average":{"1min":3.335,"5min":12.461,"15min":6.266},"min":{"1min":2.4980000000000002,"5min":2.121,"15min":1.052},"max":{"1min":4.5529999999999999,"5min":954.96299999999997,"15min":954.96299999999997},"last":3.266}]},{"osd":5,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.7250000000000001,"5min":2.5819999999999999,"15min":4.1840000000000002},"min":{"1min":1.675,"5min":1.5349999999999999,"15min":1.173},"max":{"1min":3.9769999999999999,"5min":3.9769999999999999,"15min":436.06900000000002},"last":3.9769999999999999},{"interface":"front","average":{"1min":3.1739999999999999,"5min":6.9080000000000004,"15min":4.6369999999999996},"min":{"1min":2.0840000000000001,"5min":1.8919999999999999,"15min":1.1990000000000001},"max":{"1min":4.3899999999999997,"5min":425.92899999999997,"15min":425.92899999999997},"last":2.4180000000000001}]},{"osd":6,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.96,"5min":2.7599999999999998,"15min":5.3079999999999998},"min":{"1min":2.0219999999999998,"5min":1.855,"15min":1.486},"max":{"1min":3.919,"5min":4.306,"15min":431.327},"last":3.919},{"interface":"front","average":{"1min":3.4300000000000002,"5min":3.2599999999999998,"15min":6.3230000000000004},"min":{"1min":2.5299999999999998,"5min":2.2530000000000001,"15min":1.3879999999999999},"max":{"1min":4.476,"5min":4.476,"15min":800.98800000000006},"last":3.2229999999999999}]},{"osd":7,"last update":"Tue Oct 27 04:32:03 2020","interfaces":[{"interface":"back","average":{"1min":2.8530000000000002,"5min":6.6070000000000002,"15min":12.765000000000001},"min":{"1min":2.1960000000000002,"5min":1.976,"15min":1.2789999999999999},"max":{"1min":3.8100000000000001,"5min":448.084,"15min":970.05899999999997},"last":2.4580000000000002},{"interface":"front","average":{"1min":3.1030000000000002,"5min":2.9580000000000002,"15min":5.1280000000000001},"min":{"1min":1.913,"5min":1.913,"15min":1.609},"max":{"1min":4.016,"5min":4.3490000000000002,"15min":498.803},"last":4.016}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1064984,"kb_used_data":16400,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93306856,"statfs":{"total":96636764160,"available":95546220544,"internally_reserved":1073741824,"allocated":16793600,"data_stored":16954654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":53.104999999999997,"5min":13.978999999999999,"15min":11.644},"min":{"1min":1.3340000000000001,"5min":1.1830000000000001,"15min":1.1200000000000001},"max":{"1min":924.39700000000005,"5min":924.39700000000005,"15min":924.39700000000005},"last":3.032},{"interface":"front","average":{"1min":2.355,"5min":2.2330000000000001,"15min":6.7709999999999999},"min":{"1min":1.52,"5min":0.879,"15min":0.68500000000000005},"max":{"1min":2.984,"5min":5.2169999999999996,"15min":948.09299999999996},"last":1.7849999999999999}]},{"osd":1,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.3679999999999999,"5min":14.672000000000001,"15min":6.3040000000000003},"min":{"1min":1.569,"5min":1.123,"15min":0.84699999999999998},"max":{"1min":3.335,"5min":594.51700000000005,"15min":594.51700000000005},"last":1.6739999999999999},{"interface":"front","average":{"1min":2.5619999999999998,"5min":4.0250000000000004,"15min":6.5709999999999997},"min":{"1min":1.7549999999999999,"5min":1.167,"15min":0.95299999999999996},"max":{"1min":3.0720000000000001,"5min":158.51599999999999,"15min":879.13699999999994},"last":2.4180000000000001}]},{"osd":3,"last update":"Tue Oct 27 04:31:28 2020","interfaces":[{"interface":"back","average":{"1min":2.5150000000000001,"5min":2.4409999999999998,"15min":5.0830000000000002},"min":{"1min":1.635,"5min":1.2729999999999999,"15min":1.2729999999999999},"max":{"1min":3.4830000000000001,"5min":3.6869999999999998,"15min":615.34500000000003},"last":1.7410000000000001},{"interface":"front","average":{"1min":55.475999999999999,"5min":13.064,"15min":6.0110000000000001},"min":{"1min":1.79,"5min":1.1930000000000001,"15min":1.1930000000000001},"max":{"1min":902.85299999999995,"5min":902.85299999999995,"15min":902.85299999999995},"last":2.3679999999999999}]},{"osd":4,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.2090000000000001,"5min":2.359,"15min":4.774},"min":{"1min":1.732,"5min":1.54,"15min":1.329},"max":{"1min":3.3359999999999999,"5min":3.766,"15min":551.82500000000005},"last":3.9729999999999999},{"interface":"front","average":{"1min":2.7890000000000001,"5min":19.233000000000001,"15min":14.946},"min":{"1min":1.7669999999999999,"5min":1.7470000000000001,"15min":1.248},"max":{"1min":4.3780000000000001,"5min":953.96799999999996,"15min":953.96799999999996},"last":2.4929999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9790000000000001,"5min":8.7840000000000007,"15min":15.339},"min":{"1min":1.986,"5min":1.7410000000000001,"15min":1.282},"max":{"1min":4.476,"5min":504.077,"15min":975.08900000000006},"last":2.5739999999999998},{"interface":"front","average":{"1min":3.0169999999999999,"5min":12.907999999999999,"15min":11.02},"min":{"1min":2,"5min":1.6439999999999999,"15min":1.518},"max":{"1min":3.9870000000000001,"5min":504.51799999999997,"15min":504.51799999999997},"last":2.774}]},{"osd":6,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":2.7069999999999999,"15min":8.0600000000000005},"min":{"1min":1.821,"5min":1.6890000000000001,"15min":1.506},"max":{"1min":3.427,"5min":3.871,"15min":872.27499999999998},"last":4.04},{"interface":"front","average":{"1min":3.1259999999999999,"5min":3.044,"15min":2.9540000000000002},"min":{"1min":2.2679999999999998,"5min":1.9419999999999999,"15min":1.669},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.5350000000000001}]},{"osd":7,"last update":"Tue Oct 27 04:31:11 2020","interfaces":[{"interface":"back","average":{"1min":2.9710000000000001,"5min":10.723000000000001,"15min":6.4370000000000003},"min":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":1.7450000000000001},"max":{"1min":3.891,"5min":510.29599999999999,"15min":510.29599999999999},"last":4.0899999999999999},{"interface":"front","average":{"1min":3.2989999999999999,"5min":3.1699999999999999,"15min":9.0950000000000006},"min":{"1min":2.706,"5min":2.2109999999999999,"15min":1.504},"max":{"1min":3.831,"5min":5.2839999999999998,"15min":547.11300000000006},"last":2.8210000000000002}]}]},{"osd":3,"up_from":69,"seq":296352743582,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"kb":94371840,"kb_used":1073196,"kb_used_data":24612,"kb_used_omap":7,"kb_used_meta":1048568,"kb_avail":93298644,"statfs":{"total":96636764160,"available":95537811456,"internally_reserved":1073741824,"allocated":25202688,"data_stored":26366732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8030,"internal_metadata":1073733794},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":8.8119999999999994,"15min":11.82},"min":{"1min":1.286,"5min":1.014,"15min":1.014},"max":{"1min":2.7869999999999999,"5min":706.96100000000001,"15min":938.43399999999997},"last":2.2669999999999999},{"interface":"front","average":{"1min":44.875999999999998,"5min":19.175000000000001,"15min":7.8029999999999999},"min":{"1min":1.446,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":896.04899999999998,"5min":896.04899999999998,"15min":896.04899999999998},"last":1.915}]},{"osd":1,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":22.134,"5min":6.2039999999999997,"15min":3.9359999999999999},"min":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.069},"max":{"1min":423.48000000000002,"5min":423.48000000000002,"15min":423.48000000000002},"last":2.399},{"interface":"front","average":{"1min":2.4220000000000002,"5min":2.3679999999999999,"15min":5.79},"min":{"1min":1.5089999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":3.3540000000000001,"5min":3.472,"15min":678.43899999999996},"last":2.931}]},{"osd":2,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.5699999999999998,"5min":2.4590000000000001,"15min":5.2720000000000002},"min":{"1min":1.6379999999999999,"5min":1.3620000000000001,"15min":1.194},"max":{"1min":3.4199999999999999,"5min":3.7879999999999998,"15min":971.47500000000002},"last":613.66600000000005},{"interface":"front","average":{"1min":5.5800000000000001,"5min":3.29,"15min":2.867},"min":{"1min":2.1019999999999999,"5min":1.25,"15min":1.143},"max":{"1min":63.442,"5min":63.442,"15min":63.442},"last":1.871}]},{"osd":4,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":31.300999999999998,"5min":16.239999999999998,"15min":12.332000000000001},"min":{"1min":1.9990000000000001,"5min":1.381,"15min":1.381},"max":{"1min":604.30799999999999,"5min":792.40700000000004,"15min":955.47699999999998},"last":2.468},{"interface":"front","average":{"1min":2.5899999999999999,"5min":4.0339999999999998,"15min":3.0310000000000001},"min":{"1min":1.738,"5min":1.073,"15min":1.073},"max":{"1min":3.556,"5min":151.703,"15min":151.703},"last":2.6699999999999999}]},{"osd":5,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":36.460999999999999,"5min":9.4410000000000007,"15min":7.3220000000000001},"min":{"1min":1.8859999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"max":{"1min":710.23599999999999,"5min":710.23599999999999,"15min":710.23599999999999},"last":2.141},{"interface":"front","average":{"1min":15.246,"5min":5.4690000000000003,"15min":7.5579999999999998},"min":{"1min":1.962,"5min":1.554,"15min":1.554},"max":{"1min":261.73599999999999,"5min":261.73599999999999,"15min":607.42600000000004},"last":2.9740000000000002}]},{"osd":6,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":28.733000000000001,"5min":13.307,"15min":6.734},"min":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.55},"max":{"1min":522.255,"5min":522.255,"15min":522.255},"last":2.3610000000000002},{"interface":"front","average":{"1min":2.9729999999999999,"5min":3.0019999999999998,"15min":2.9929999999999999},"min":{"1min":2.2719999999999998,"5min":1.772,"15min":1.772},"max":{"1min":3.8439999999999999,"5min":4.3899999999999997,"15min":4.3899999999999997},"last":3.0339999999999998}]},{"osd":7,"last update":"Tue Oct 27 04:31:43 2020","interfaces":[{"interface":"back","average":{"1min":2.9849999999999999,"5min":12.74,"15min":8.8989999999999991},"min":{"1min":2.1600000000000001,"5min":2.0979999999999999,"15min":1.667},"max":{"1min":3.6429999999999998,"5min":607.92200000000003,"15min":867.16300000000001},"last":2.617},{"interface":"front","average":{"1min":3.0920000000000001,"5min":16.859999999999999,"15min":9.5329999999999995},"min":{"1min":2.0529999999999999,"5min":1.5960000000000001,"15min":1.3100000000000001},"max":{"1min":3.8279999999999998,"5min":746.22299999999996,"15min":746.22299999999996},"last":2.5640000000000001}]}]}]}} Ignoring malparsed backtrace: instructing pg 2.ds0 on osd.5 to deep-scrub , 2020-10-27T04:32:32.200 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. , 2020-10-27T04:32:52.202 INFO:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph pg dump --format=json Ignoring malparsed backtrace: 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.9 last_scrub_stamp 2020-10-27 04:19:13.189245 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:33:34.376 INFO:tasks.ceph:pgid 1.8 last_scrub_stamp 2020-10-27 04:19:01.325624 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=1, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:34:36.452 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2020-10-27 04:18:55.292162 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=55, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.662 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:35:18.663 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2020-10-27 04:17:46.831755 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=17, tm_sec=46, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2020-10-27 04:18:58.356124 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=58, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.770 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2020-10-27 04:18:13.242061 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=13, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) , 2020-10-27T04:36:20.771 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2020-10-27 04:18:52.955136 time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=18, tm_sec=52, tm_wday=1, tm_yday=301, tm_isdst=-1) <= time.struct_time(tm_year=2020, tm_mon=10, tm_mday=27, tm_hour=4, tm_min=19, tm_sec=49, tm_wday=1, tm_yday=301, tm_isdst=0) Ignoring malparsed backtrace: 2020-10-27T04:36:41.487 DEBUG:tasks.ceph.osd.1:waiting for process to exit , 2020-10-27T04:36:41.488 INFO:teuthology.orchestra.run:waiting for 300 , 2020-10-27T04:36:41.522 INFO:tasks.ceph.osd.1:Stopped Ignoring malparsed backtrace: 2020-10-27T04:36:58.697 ERROR:teuthology.run_tasks:Manager failed: ceph , Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_ceph-c_nautilus-testing-1027/qa/tasks/ceph.py", line 1873, in task Ignoring malparsed backtrace: 2020-10-27T04:37:00.234 INFO:teuthology.task.internal.syslog:Checking logs for errors... , 2020-10-27T04:37:00.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com , 2020-10-27T04:37:00.235 INFO:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 Found 3 distinct failure reasons Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' 2 jobs: ['5563939', '5563940'] suites intersection: ['clusters/{fixed-2.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'thrashosds-health.yaml'] suites union: ['clusters/{fixed-2.yaml', 'fast/fast.yaml', 'fast/normal.yaml', 'msgr-failures/few.yaml', 'msgr-failures/osd-delay.yaml', 'objectstore/bluestore-bitmap.yaml', 'objectstore/bluestore-stupid.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'recovery-overrides/{more-async-partial-recovery.yaml}', 'recovery-overrides/{more-partial-recovery.yaml}', 'supported-random-distro$/{centos_latest.yaml}', 'supported-random-distro$/{ubuntu_16.04.yaml}', 'thrashers/default.yaml', 'thrashers/fastread.yaml', 'thrashosds-health.yaml', 'workloads/ec-small-objects-fast-read.yaml}', 'workloads/ec-small-objects-many-deletes.yaml}'] Dead: b'2020-10-27T15:08:57.960 DEBUG:teuthology.exit:Finished running handlers' 4 jobs: ['5563936', '5563941', '5563942', '5563937'] suites intersection: ['clusters/{fixed-2.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'thrashosds-health.yaml'] suites union: ['clusters/{fixed-2.yaml', 'fast/fast.yaml', 'fast/normal.yaml', 'msgr-failures/fastclose.yaml', 'msgr-failures/few.yaml', 'msgr-failures/osd-delay.yaml', 'objectstore/bluestore-comp-lz4.yaml', 'objectstore/bluestore-comp-snappy.yaml', 'objectstore/bluestore-comp-zlib.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'recovery-overrides/{default.yaml}', 'recovery-overrides/{more-active-recovery.yaml}', 'recovery-overrides/{more-async-partial-recovery.yaml}', 'recovery-overrides/{more-async-recovery.yaml}', 'supported-random-distro$/{centos_latest.yaml}', 'supported-random-distro$/{ubuntu_latest.yaml}', 'thrashers/morepggrow.yaml', 'thrashers/pggrow.yaml', 'thrashosds-health.yaml', 'workloads/ec-rados-plugin=clay-k=4-m=2.yaml}', 'workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}', 'workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml}', 'workloads/ec-small-objects.yaml}'] Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0' 1 jobs: ['5563943'] suites: ['clusters/{fixed-2.yaml', 'fast/fast.yaml', 'msgr-failures/osd-delay.yaml', 'objectstore/bluestore-comp-zlib.yaml', 'openstack.yaml}', 'rados.yaml', 'rados:thrash-erasure-code/{ceph.yaml', 'recovery-overrides/{default.yaml}', 'supported-random-distro$/{centos_latest.yaml}', 'thrashers/careful.yaml', 'thrashosds-health.yaml', 'workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml}']