2021-03-13T09:38:13.929 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-13T09:38:13.931 INFO:teuthology.results:subset: '10/5000' 2021-03-13T09:38:13.931 INFO:teuthology.results:seed: '3931' 2021-03-14T01:39:28.666 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-14T01:39:45.151 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-03-14T06:22:13.506 INFO:teuthology.results:Tests finished! gathering results... 2021-03-14T06:23:21.845 INFO:scrape:Found 522 jobs 2021-03-14T06:23:22.091 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f379bed15dd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f379bed17b8] 2021-03-14T06:23:22.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.578 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.886925+0000", , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0041089899999999999 , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:258 3.10s0 3:0e12bfe6:::gibba01223948-25:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected e37)", 2021-03-14T06:23:22.093 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.629 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.870591+0000", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0 2021-03-14T06:23:22.093 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "dispatched", , 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.926358+0000", , 2021-03-14T01:35:47.655 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 1.5138e-05 2021-03-14T06:23:22.095 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.676 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.973849+0000", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0011102919999999999 2021-03-14T06:23:22.095 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.847121+0000", 2021-03-14T06:23:22.096 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", , 2021-03-14T01:35:47.724 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.851310+0000", 2021-03-14T06:23:22.097 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "sub_op_started", , 2021-03-14T01:35:47.747 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.852683+0000", 2021-03-14T06:23:22.098 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.772 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", 2021-03-14T06:23:22.099 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", 2021-03-14T06:23:22.100 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 6.9084000000000005e-05 , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.100 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:18.185620+0000", , 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015103299999999999 , 2021-03-14T01:35:47.845 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.101 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:37.118351+0000", , 2021-03-14T01:35:47.870 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00021992299999999999 2021-03-14T06:23:22.102 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.104 INFO:teuthology.orchestra.run.gibba019.stdout: "complaint_time": 30, , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout:} 2021-03-14T06:23:22.102 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.224 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00011459900000000001 , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.103 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0010301640000000001 , 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.247 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.103 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:227 3.13s0 3:cbacc145:::gibba01223948-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", , 2021-03-14T01:35:48.269 INFO:teuthology.orchestra.run.gibba019.stdout: "initiated_at": "2021-03-14T01:34:30.853680+0000", 2021-03-14T06:23:22.104 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.290 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.865509+0000", , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00091812100000000004 , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.868438+0000", , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 4.1616000000000003e-05 , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.037303+0000", , 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.16912944399999999 , 2021-03-14T01:35:48.335 INFO:teuthology.orchestra.run.gibba019.stdout: } 2021-03-14T06:23:22.106 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "all_read", , 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.878076+0000", , 2021-03-14T01:35:48.357 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.15851783999999999 2021-03-14T06:23:22.107 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.935113+0000", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015084699999999999 2021-03-14T06:23:22.107 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "age": 75.293399738000005, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.023467689999999999, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "type_data": { 2021-03-14T06:23:22.108 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.916508+0000", 2021-03-14T06:23:22.109 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.443 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.890695+0000", 2021-03-14T06:23:22.109 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.465 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.902943+0000", 2021-03-14T06:23:22.110 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: "events": [ , 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.492 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", 2021-03-14T06:23:22.110 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.513 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", 2021-03-14T06:23:22.111 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "done", 2021-03-14T06:23:22.112 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.868 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "throttled", , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:45.889084+0000", 2021-03-14T06:23:22.112 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.893 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:46.991105+0000", , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 61.871998275999999, , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.015193409999999999, 2021-03-14T06:23:22.113 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.918 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.114 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", 2021-03-14T06:23:22.114 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.966 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "started", , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client_info": { , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client": "client.4440", 2021-03-14T06:23:22.115 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.017557843 , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.115 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:22.894694+0000", , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.116 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.333651+0000", , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 3.9671e-05 , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.117 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", , 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.896834+0000", , 2021-03-14T01:35:49.064 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 1.3286e-05 2021-03-14T06:23:22.117 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.043394+0000", 2021-03-14T06:23:22.118 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.112 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.063994+0000", 2021-03-14T06:23:22.119 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.138 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.119 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "reached pg", , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "events": [ , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.120 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.120 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.521 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 8.9042000000000006e-05 , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.121 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000109078 , 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.545 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.122 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0030517500000000002 , 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.568 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.122 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "description": "osd_op(client.4440.0:845 3.13s0 3:c9b54c63:::gibba01223948-48:head [read 0~802816,checksum] snapc 0=[] ondisk+read+known_if_redirected e81)", , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:35:17.019254+0000", 2021-03-14T06:23:22.123 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.610 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019324+0000", , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.9320000000000006e-05 , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.124 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019543+0000", , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 4.5383999999999998e-05 , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.125 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019697+0000", , 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7690000000000001e-06 , 2021-03-14T01:35:49.654 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.126 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019750+0000", , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7429999999999998e-06 , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.126 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.856141+0000", , 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.6579999999999999e-06 , 2021-03-14T01:35:49.697 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.127 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.849415+0000", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.8174000000000002e-05 2021-03-14T06:23:22.127 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:29.483870+0000", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.61419095999999995 2021-03-14T06:23:22.128 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.760 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.890039+0000", 2021-03-14T06:23:22.129 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.130 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.026 INFO:teuthology.orchestra.run.gibba037.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:50.027 INFO:teuthology.orchestra.run.gibba037.stdout:} , 2021-03-14T01:35:50.034 DEBUG:teuthology.orchestra.run.gibba037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2021-03-14T06:23:22.130 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.154 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "reached_pg", , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:46.952878+0000", 2021-03-14T06:23:22.131 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.878648+0000", 2021-03-14T06:23:22.132 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.888915+0000", 2021-03-14T06:23:22.132 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.222 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.133 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.133 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.1989999999999997e-06 , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.134 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000195164 , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.135 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:12.876823+0000", , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.136 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:13.696049+0000", , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.79497452899999999 , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.136 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.056841+0000", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 2021-03-14T06:23:22.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.846950+0000", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024717700000000001 2021-03-14T06:23:22.137 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.868074+0000", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.2250999999999997e-05 2021-03-14T06:23:22.138 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.420 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.047676+0000", 2021-03-14T06:23:22.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.751 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 600, , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: "ops": [ , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.139 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.774 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:27.518527+0000", 2021-03-14T06:23:22.140 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:27.521467+0000", , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 83.228323915999994, , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.145138142, 2021-03-14T06:23:22.140 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.822 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.141 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.447e-06 , 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.846 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.868 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0051480689999999999 , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.142 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 6.614e-05 , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.143 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024514099999999999 , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:33.493203+0000", , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00014740399999999999 , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.144 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.044002+0000", , 2021-03-14T01:35:50.957 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0061366989999999998 2021-03-14T06:23:22.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.979 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 3.7041930550000002, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.033522228000000001, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "type_data": { 2021-03-14T06:23:22.145 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.001 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.058774+0000", 2021-03-14T06:23:22.146 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.147 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.045 INFO:teuthology.orchestra.run.gibba037.stdout:} , marked in osd.2. , nodeep-scrub is unset 2021-03-14T06:23:22.150 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:19.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats , 2021-03-14T01:36:19.816 INFO:teuthology.orchestra.run.gibba012.stdout:73014444063 , 2021-03-14T01:36:19.817 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2021-03-14T06:23:22.151 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:27.369 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout:{ , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-03-14T06:23:22.154 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-03-14T01:35:55.615131+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=55, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-14T01:35:41.258402+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=41, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.239 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-14T01:36:14.843139+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=14, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) 2021-03-14T06:23:22.155 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 265, in thread , run.wait(tests.values()) 2021-03-14T06:23:22.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:37:46.578 DEBUG:teuthology.orchestra.run.gibba012:> mktemp , 2021-03-14T01:37:46.586 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.woWAIkQTGM , 2021-03-14T01:37:46.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.woWAIkQTGM 2021-03-14T06:23:22.156 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 177 0.100 1.955 1.112 , 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv04.front.sepi 96.245.170.99 3 u 12 64 177 0.051 0.756 1.323 , 2021-03-14T01:38:02.276 DEBUG:teuthology.orchestra.run.gibba037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-14T06:23:22.157 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2021-03-14T06:23:22.254 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f379bed15dd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f379bed17b8] 2021-03-14T06:23:22.255 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.578 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.886925+0000", , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0041089899999999999 , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.255 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:258 3.10s0 3:0e12bfe6:::gibba01223948-25:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected e37)", 2021-03-14T06:23:22.256 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.629 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.870591+0000", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0 2021-03-14T06:23:22.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "dispatched", , 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.926358+0000", , 2021-03-14T01:35:47.655 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 1.5138e-05 2021-03-14T06:23:22.257 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.676 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.973849+0000", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0011102919999999999 2021-03-14T06:23:22.258 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.847121+0000", 2021-03-14T06:23:22.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", , 2021-03-14T01:35:47.724 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.851310+0000", 2021-03-14T06:23:22.259 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "sub_op_started", , 2021-03-14T01:35:47.747 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.852683+0000", 2021-03-14T06:23:22.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.772 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", 2021-03-14T06:23:22.260 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", 2021-03-14T06:23:22.261 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 6.9084000000000005e-05 , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.262 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:18.185620+0000", , 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015103299999999999 , 2021-03-14T01:35:47.845 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.262 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:37.118351+0000", , 2021-03-14T01:35:47.870 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00021992299999999999 2021-03-14T06:23:22.263 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.104 INFO:teuthology.orchestra.run.gibba019.stdout: "complaint_time": 30, , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout:} 2021-03-14T06:23:22.263 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.224 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00011459900000000001 , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.264 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0010301640000000001 , 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.247 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.265 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:227 3.13s0 3:cbacc145:::gibba01223948-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", , 2021-03-14T01:35:48.269 INFO:teuthology.orchestra.run.gibba019.stdout: "initiated_at": "2021-03-14T01:34:30.853680+0000", 2021-03-14T06:23:22.265 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.290 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.865509+0000", , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00091812100000000004 , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.266 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.868438+0000", , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 4.1616000000000003e-05 , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.266 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.037303+0000", , 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.16912944399999999 , 2021-03-14T01:35:48.335 INFO:teuthology.orchestra.run.gibba019.stdout: } 2021-03-14T06:23:22.267 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "all_read", , 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.878076+0000", , 2021-03-14T01:35:48.357 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.15851783999999999 2021-03-14T06:23:22.268 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.935113+0000", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015084699999999999 2021-03-14T06:23:22.268 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "age": 75.293399738000005, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.023467689999999999, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "type_data": { 2021-03-14T06:23:22.269 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.916508+0000", 2021-03-14T06:23:22.270 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.443 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.890695+0000", 2021-03-14T06:23:22.270 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.465 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.902943+0000", 2021-03-14T06:23:22.271 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: "events": [ , 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.492 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", 2021-03-14T06:23:22.271 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.513 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", 2021-03-14T06:23:22.272 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "done", 2021-03-14T06:23:22.273 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.868 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "throttled", , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:45.889084+0000", 2021-03-14T06:23:22.273 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.893 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:46.991105+0000", , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 61.871998275999999, , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.015193409999999999, 2021-03-14T06:23:22.274 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.918 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.274 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", 2021-03-14T06:23:22.275 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.966 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "started", , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client_info": { , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client": "client.4440", 2021-03-14T06:23:22.276 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.017557843 , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.276 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:22.894694+0000", , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.277 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.333651+0000", , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 3.9671e-05 , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.277 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", , 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.896834+0000", , 2021-03-14T01:35:49.064 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 1.3286e-05 2021-03-14T06:23:22.278 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.043394+0000", 2021-03-14T06:23:22.279 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.112 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.063994+0000", 2021-03-14T06:23:22.279 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.138 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.280 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "reached pg", , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "events": [ , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.280 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.281 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.521 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 8.9042000000000006e-05 , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.282 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000109078 , 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.545 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.282 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0030517500000000002 , 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.568 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.283 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "description": "osd_op(client.4440.0:845 3.13s0 3:c9b54c63:::gibba01223948-48:head [read 0~802816,checksum] snapc 0=[] ondisk+read+known_if_redirected e81)", , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:35:17.019254+0000", 2021-03-14T06:23:22.283 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.610 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019324+0000", , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.9320000000000006e-05 , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.284 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019543+0000", , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 4.5383999999999998e-05 , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.285 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019697+0000", , 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7690000000000001e-06 , 2021-03-14T01:35:49.654 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.285 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019750+0000", , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7429999999999998e-06 , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.286 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.856141+0000", , 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.6579999999999999e-06 , 2021-03-14T01:35:49.697 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.287 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.849415+0000", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.8174000000000002e-05 2021-03-14T06:23:22.287 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:29.483870+0000", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.61419095999999995 2021-03-14T06:23:22.288 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.760 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.890039+0000", 2021-03-14T06:23:22.288 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.289 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.026 INFO:teuthology.orchestra.run.gibba037.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:50.027 INFO:teuthology.orchestra.run.gibba037.stdout:} , 2021-03-14T01:35:50.034 DEBUG:teuthology.orchestra.run.gibba037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2021-03-14T06:23:22.290 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.154 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "reached_pg", , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:46.952878+0000", 2021-03-14T06:23:22.290 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.878648+0000", 2021-03-14T06:23:22.291 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.888915+0000", 2021-03-14T06:23:22.291 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.222 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.292 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.293 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.1989999999999997e-06 , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.293 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000195164 , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.294 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:12.876823+0000", , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.294 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:13.696049+0000", , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.79497452899999999 , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.295 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.056841+0000", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 2021-03-14T06:23:22.296 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.846950+0000", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024717700000000001 2021-03-14T06:23:22.296 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.868074+0000", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.2250999999999997e-05 2021-03-14T06:23:22.297 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.420 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.047676+0000", 2021-03-14T06:23:22.297 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.751 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 600, , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: "ops": [ , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.298 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.774 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:27.518527+0000", 2021-03-14T06:23:22.299 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:27.521467+0000", , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 83.228323915999994, , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.145138142, 2021-03-14T06:23:22.299 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.822 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.300 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.447e-06 , 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.846 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.300 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.868 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0051480689999999999 , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.301 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 6.614e-05 , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.302 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024514099999999999 , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.302 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:33.493203+0000", , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00014740399999999999 , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.303 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.044002+0000", , 2021-03-14T01:35:50.957 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0061366989999999998 2021-03-14T06:23:22.303 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.979 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 3.7041930550000002, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.033522228000000001, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "type_data": { 2021-03-14T06:23:22.304 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.001 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.058774+0000", 2021-03-14T06:23:22.305 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.305 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.045 INFO:teuthology.orchestra.run.gibba037.stdout:} , marked in osd.2. , nodeep-scrub is unset 2021-03-14T06:23:22.309 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:19.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats , 2021-03-14T01:36:19.816 INFO:teuthology.orchestra.run.gibba012.stdout:73014444063 , 2021-03-14T01:36:19.817 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2021-03-14T06:23:22.310 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:27.369 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout:{ , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-03-14T06:23:22.313 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-03-14T01:35:55.615131+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=55, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-14T01:35:41.258402+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=41, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.239 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-14T01:36:14.843139+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=14, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) 2021-03-14T06:23:22.313 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 265, in thread , run.wait(tests.values()) 2021-03-14T06:23:22.314 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:37:46.578 DEBUG:teuthology.orchestra.run.gibba012:> mktemp , 2021-03-14T01:37:46.586 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.woWAIkQTGM , 2021-03-14T01:37:46.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.woWAIkQTGM 2021-03-14T06:23:22.315 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 177 0.100 1.955 1.112 , 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv04.front.sepi 96.245.170.99 3 u 12 64 177 0.051 0.756 1.323 , 2021-03-14T01:38:02.276 DEBUG:teuthology.orchestra.run.gibba037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-14T06:23:22.315 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2021-03-14T06:23:22.407 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7f379bed15dd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7f379bed17b8] 2021-03-14T06:23:22.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.578 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.886925+0000", , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0041089899999999999 , 2021-03-14T01:35:47.579 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.606 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:258 3.10s0 3:0e12bfe6:::gibba01223948-25:head [setxattr _header (55) in=7b] snapc 5=[2] ondisk+write+known_if_redirected e37)", 2021-03-14T06:23:22.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.629 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.870591+0000", , 2021-03-14T01:35:47.630 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0 2021-03-14T06:23:22.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "dispatched", , 2021-03-14T01:35:47.654 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.926358+0000", , 2021-03-14T01:35:47.655 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 1.5138e-05 2021-03-14T06:23:22.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.676 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.973849+0000", , 2021-03-14T01:35:47.677 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0011102919999999999 2021-03-14T06:23:22.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.702 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.847121+0000", 2021-03-14T06:23:22.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.723 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", , 2021-03-14T01:35:47.724 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.851310+0000", 2021-03-14T06:23:22.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.746 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "sub_op_started", , 2021-03-14T01:35:47.747 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:41.852683+0000", 2021-03-14T06:23:22.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.772 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.773 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", 2021-03-14T06:23:22.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:47.794 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "started", 2021-03-14T06:23:22.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 6.9084000000000005e-05 , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:47.821 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:18.185620+0000", , 2021-03-14T01:35:47.844 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015103299999999999 , 2021-03-14T01:35:47.845 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:47.869 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:35:37.118351+0000", , 2021-03-14T01:35:47.870 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00021992299999999999 2021-03-14T06:23:22.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.104 INFO:teuthology.orchestra.run.gibba019.stdout: "complaint_time": 30, , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:48.105 INFO:teuthology.orchestra.run.gibba019.stdout:} 2021-03-14T06:23:22.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.224 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00011459900000000001 , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.225 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.0010301640000000001 , 2021-03-14T01:35:48.246 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.247 INFO:teuthology.orchestra.run.gibba019.stdout: { 2021-03-14T06:23:22.418 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.268 INFO:teuthology.orchestra.run.gibba019.stdout: "description": "osd_op(client.4440.0:227 3.13s0 3:cbacc145:::gibba01223948-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e35)", , 2021-03-14T01:35:48.269 INFO:teuthology.orchestra.run.gibba019.stdout: "initiated_at": "2021-03-14T01:34:30.853680+0000", 2021-03-14T06:23:22.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.290 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.865509+0000", , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00091812100000000004 , 2021-03-14T01:35:48.291 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.868438+0000", , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 4.1616000000000003e-05 , 2021-03-14T01:35:48.313 INFO:teuthology.orchestra.run.gibba019.stdout: }, 2021-03-14T06:23:22.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.037303+0000", , 2021-03-14T01:35:48.334 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.16912944399999999 , 2021-03-14T01:35:48.335 INFO:teuthology.orchestra.run.gibba019.stdout: } 2021-03-14T06:23:22.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "all_read", , 2021-03-14T01:35:48.356 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:31.878076+0000", , 2021-03-14T01:35:48.357 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.15851783999999999 2021-03-14T06:23:22.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.935113+0000", , 2021-03-14T01:35:48.378 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.00015084699999999999 2021-03-14T06:23:22.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "age": 75.293399738000005, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "duration": 0.023467689999999999, , 2021-03-14T01:35:48.400 INFO:teuthology.orchestra.run.gibba019.stdout: "type_data": { 2021-03-14T06:23:22.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "throttled", , 2021-03-14T01:35:48.422 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:32.916508+0000", 2021-03-14T06:23:22.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.443 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:48.444 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.890695+0000", 2021-03-14T06:23:22.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.465 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "commit_sent", , 2021-03-14T01:35:48.466 INFO:teuthology.orchestra.run.gibba019.stdout: "time": "2021-03-14T01:34:35.902943+0000", 2021-03-14T06:23:22.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: "events": [ , 2021-03-14T01:35:48.491 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.492 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "initiated", 2021-03-14T06:23:22.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.513 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.514 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "reached_pg", 2021-03-14T06:23:22.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: }, , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: { , 2021-03-14T01:35:48.536 INFO:teuthology.orchestra.run.gibba019.stdout: "event": "done", 2021-03-14T06:23:22.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.868 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "throttled", , 2021-03-14T01:35:48.869 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:45.889084+0000", 2021-03-14T06:23:22.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.893 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:46.991105+0000", , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 61.871998275999999, , 2021-03-14T01:35:48.894 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.015193409999999999, 2021-03-14T06:23:22.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.918 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.919 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:48.943 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", 2021-03-14T06:23:22.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.966 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "started", , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client_info": { , 2021-03-14T01:35:48.967 INFO:teuthology.orchestra.run.gibba037.stdout: "client": "client.4440", 2021-03-14T06:23:22.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.017557843 , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:48.990 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:22.894694+0000", , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:49.014 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.333651+0000", , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 3.9671e-05 , 2021-03-14T01:35:49.039 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", , 2021-03-14T01:35:49.063 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.896834+0000", , 2021-03-14T01:35:49.064 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 1.3286e-05 2021-03-14T06:23:22.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:49.088 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.043394+0000", 2021-03-14T06:23:22.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.112 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:49.113 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.063994+0000", 2021-03-14T06:23:22.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.137 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.138 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "flag_point": "reached pg", , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: "events": [ , 2021-03-14T01:35:49.161 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.498 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", 2021-03-14T06:23:22.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.521 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 8.9042000000000006e-05 , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.522 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000109078 , 2021-03-14T01:35:49.544 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.545 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0030517500000000002 , 2021-03-14T01:35:49.567 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.568 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "description": "osd_op(client.4440.0:845 3.13s0 3:c9b54c63:::gibba01223948-48:head [read 0~802816,checksum] snapc 0=[] ondisk+read+known_if_redirected e81)", , 2021-03-14T01:35:49.589 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:35:17.019254+0000", 2021-03-14T06:23:22.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.610 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019324+0000", , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.9320000000000006e-05 , 2021-03-14T01:35:49.611 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019543+0000", , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 4.5383999999999998e-05 , 2021-03-14T01:35:49.632 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019697+0000", , 2021-03-14T01:35:49.653 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7690000000000001e-06 , 2021-03-14T01:35:49.654 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:17.019750+0000", , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.7429999999999998e-06 , 2021-03-14T01:35:49.675 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.856141+0000", , 2021-03-14T01:35:49.696 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.6579999999999999e-06 , 2021-03-14T01:35:49.697 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.849415+0000", , 2021-03-14T01:35:49.718 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 2.8174000000000002e-05 2021-03-14T06:23:22.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:29.483870+0000", , 2021-03-14T01:35:49.739 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.61419095999999995 2021-03-14T06:23:22.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.760 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:49.761 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:41.890039+0000", 2021-03-14T06:23:22.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:49.782 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.026 INFO:teuthology.orchestra.run.gibba037.stdout: "num_blocked_ops": 0 , 2021-03-14T01:35:50.027 INFO:teuthology.orchestra.run.gibba037.stdout:} , 2021-03-14T01:35:50.034 DEBUG:teuthology.orchestra.run.gibba037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2021-03-14T06:23:22.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.154 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "reached_pg", , 2021-03-14T01:35:50.155 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:46.952878+0000", 2021-03-14T06:23:22.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_committed", , 2021-03-14T01:35:50.178 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.878648+0000", 2021-03-14T06:23:22.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "done", , 2021-03-14T01:35:50.200 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:03.888915+0000", 2021-03-14T06:23:22.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.222 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.223 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "header_read", 2021-03-14T06:23:22.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.245 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 5.1989999999999997e-06 , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.267 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.000195164 , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.289 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:12.876823+0000", , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 , 2021-03-14T01:35:50.311 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:13.696049+0000", , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.79497452899999999 , 2021-03-14T01:35:50.333 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "initiated", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:18.056841+0000", , 2021-03-14T01:35:50.355 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0 2021-03-14T06:23:22.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "queued_for_pg", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.846950+0000", , 2021-03-14T01:35:50.377 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024717700000000001 2021-03-14T06:23:22.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:28.868074+0000", , 2021-03-14T01:35:50.399 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.2250999999999997e-05 2021-03-14T06:23:22.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.420 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "dispatched", , 2021-03-14T01:35:50.421 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.047676+0000", 2021-03-14T06:23:22.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.751 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 600, , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: "ops": [ , 2021-03-14T01:35:50.752 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:50.773 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "started", , 2021-03-14T01:35:50.774 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:34:27.518527+0000", 2021-03-14T06:23:22.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "initiated_at": "2021-03-14T01:34:27.521467+0000", , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 83.228323915999994, , 2021-03-14T01:35:50.799 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.145138142, 2021-03-14T06:23:22.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.822 INFO:teuthology.orchestra.run.gibba037.stdout: } , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: ] , 2021-03-14T01:35:50.823 INFO:teuthology.orchestra.run.gibba037.stdout: } 2021-03-14T06:23:22.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 7.447e-06 , 2021-03-14T01:35:50.845 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.846 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.868 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0051480689999999999 , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.869 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 6.614e-05 , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.892 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00024514099999999999 , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:50.913 INFO:teuthology.orchestra.run.gibba037.stdout: { 2021-03-14T06:23:22.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:33.493203+0000", , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.00014740399999999999 , 2021-03-14T01:35:50.935 INFO:teuthology.orchestra.run.gibba037.stdout: }, 2021-03-14T06:23:22.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", , 2021-03-14T01:35:50.956 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.044002+0000", , 2021-03-14T01:35:50.957 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.0061366989999999998 2021-03-14T06:23:22.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:50.979 INFO:teuthology.orchestra.run.gibba037.stdout: "age": 3.7041930550000002, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "duration": 0.033522228000000001, , 2021-03-14T01:35:50.980 INFO:teuthology.orchestra.run.gibba037.stdout: "type_data": { 2021-03-14T06:23:22.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.001 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "sub_op_started", , 2021-03-14T01:35:51.002 INFO:teuthology.orchestra.run.gibba037.stdout: "time": "2021-03-14T01:35:47.058774+0000", 2021-03-14T06:23:22.458 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: }, , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: { , 2021-03-14T01:35:51.024 INFO:teuthology.orchestra.run.gibba037.stdout: "event": "all_read", 2021-03-14T06:23:22.459 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:35:51.045 INFO:teuthology.orchestra.run.gibba037.stdout:} , marked in osd.2. , nodeep-scrub is unset 2021-03-14T06:23:22.462 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:19.677 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats , 2021-03-14T01:36:19.816 INFO:teuthology.orchestra.run.gibba012.stdout:73014444063 , 2021-03-14T01:36:19.817 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2021-03-14T06:23:22.464 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:27.369 DEBUG:teuthology.orchestra.run.gibba012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout:{ , 2021-03-14T01:36:27.525 INFO:teuthology.orchestra.run.gibba012.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-03-14T06:23:22.468 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-03-14T01:35:55.615131+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=55, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.238 INFO:tasks.ceph:pgid 1.13 last_scrub_stamp 2021-03-14T01:35:41.258402+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=35, tm_sec=41, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) , 2021-03-14T01:36:30.239 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2021-03-14T01:36:14.843139+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=14, tm_wday=6, tm_yday=73, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=14, tm_hour=1, tm_min=36, tm_sec=24, tm_wday=6, tm_yday=73, tm_isdst=0) 2021-03-14T06:23:22.472 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 265, in thread , run.wait(tests.values()) 2021-03-14T06:23:22.473 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:37:46.578 DEBUG:teuthology.orchestra.run.gibba012:> mktemp , 2021-03-14T01:37:46.586 INFO:teuthology.orchestra.run.gibba012.stdout:/tmp/tmp.woWAIkQTGM , 2021-03-14T01:37:46.587 DEBUG:teuthology.orchestra.run.gibba012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.woWAIkQTGM 2021-03-14T06:23:22.474 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 177 0.100 1.955 1.112 , 2021-03-14T01:38:02.275 INFO:teuthology.orchestra.run.gibba019.stdout:+hv04.front.sepi 96.245.170.99 3 u 12 64 177 0.051 0.756 1.323 , 2021-03-14T01:38:02.276 DEBUG:teuthology.orchestra.run.gibba037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-14T06:23:22.476 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise , raise value.with_traceback(tb) 2021-03-14T06:23:23.068 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7fec85559308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7fec85559522] 2021-03-14T06:23:23.069 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:23:59.816 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:21:36.293316+0000", , 2021-03-13T17:23:59.816 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0.00067577999999999996 , 2021-03-13T17:23:59.816 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-03-14T06:23:23.069 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:23:59.877 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0.013001581, , 2021-03-13T17:23:59.878 INFO:teuthology.orchestra.run.gibba020.stdout: "type_data": { , 2021-03-13T17:23:59.878 INFO:teuthology.orchestra.run.gibba020.stdout: "flag_point": "started", 2021-03-14T06:23:23.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:23:59.928 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "initiated", , 2021-03-13T17:23:59.928 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:02.890767+0000", , 2021-03-13T17:23:59.928 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0 2021-03-14T06:23:23.071 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:23:59.981 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "all_read", , 2021-03-13T17:23:59.982 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:02.890849+0000", , 2021-03-13T17:23:59.982 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 5.4999999999999999e-06 2021-03-14T06:23:23.072 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.039 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "queued_for_pg", , 2021-03-13T17:24:00.039 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:04.814307+0000", , 2021-03-13T17:24:00.039 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0.0036947970000000001 2021-03-14T06:23:23.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.117 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "sub_op_started", , 2021-03-13T17:24:00.118 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:04.823624+0000", , 2021-03-13T17:24:00.120 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0.0024860849999999999 2021-03-14T06:23:23.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.198 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "done", , 2021-03-13T17:24:00.198 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:04.826989+0000", , 2021-03-13T17:24:00.198 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 0.0073293899999999999 2021-03-14T06:23:23.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.245 INFO:teuthology.orchestra.run.gibba020.stdout: "client": "client.4509", , 2021-03-13T17:24:00.246 INFO:teuthology.orchestra.run.gibba020.stdout: "client_addr": "172.21.2.120:0/1217917419", , 2021-03-13T17:24:00.246 INFO:teuthology.orchestra.run.gibba020.stdout: "tid": 5178 2021-03-14T06:23:23.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.282 INFO:teuthology.orchestra.run.gibba020.stdout: { , 2021-03-13T17:24:00.282 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "all_read", , 2021-03-13T17:24:00.283 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:11.814869+0000", 2021-03-14T06:23:23.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.310 INFO:teuthology.orchestra.run.gibba020.stdout: { , 2021-03-13T17:24:00.310 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "reached_pg", , 2021-03-13T17:24:00.310 INFO:teuthology.orchestra.run.gibba020.stdout: "time": "2021-03-13T17:23:11.825137+0000", 2021-03-14T06:23:23.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.338 INFO:teuthology.orchestra.run.gibba020.stdout: ] , 2021-03-13T17:24:00.339 INFO:teuthology.orchestra.run.gibba020.stdout: } , 2021-03-13T17:24:00.339 INFO:teuthology.orchestra.run.gibba020.stdout: }, 2021-03-14T06:23:23.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.370 INFO:teuthology.orchestra.run.gibba020.stdout: }, , 2021-03-13T17:24:00.370 INFO:teuthology.orchestra.run.gibba020.stdout: { , 2021-03-13T17:24:00.370 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "throttled", 2021-03-14T06:23:23.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.455 INFO:teuthology.orchestra.run.gibba020.stdout: }, , 2021-03-13T17:24:00.455 INFO:teuthology.orchestra.run.gibba020.stdout: { , 2021-03-13T17:24:00.455 INFO:teuthology.orchestra.run.gibba020.stdout: "event": "sub_op_started", 2021-03-14T06:23:23.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.501 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 7.8269999999999998e-06 , 2021-03-13T17:24:00.501 INFO:teuthology.orchestra.run.gibba020.stdout: }, , 2021-03-13T17:24:00.501 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-03-14T06:23:23.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.551 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 1.002209041 , 2021-03-13T17:24:00.552 INFO:teuthology.orchestra.run.gibba020.stdout: }, , 2021-03-13T17:24:00.552 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-03-14T06:23:23.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:00.589 INFO:teuthology.orchestra.run.gibba020.stdout: "duration": 1.889e-06 , 2021-03-13T17:24:00.589 INFO:teuthology.orchestra.run.gibba020.stdout: }, , 2021-03-13T17:24:00.589 INFO:teuthology.orchestra.run.gibba020.stdout: { 2021-03-14T06:23:23.081 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 179, in _raise_for_status , raise CommandCrashedError(command=self.command) , teuthology.exceptions.CommandCrashedError: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0' 2021-03-14T06:23:23.082 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:17.202 INFO:teuthology.orchestra.run.gibba020.stdout:mon.a: {} , 2021-03-13T17:24:17.209 INFO:teuthology.orchestra.run.gibba020.stdout:mon.b: {} , 2021-03-13T17:24:17.216 INFO:teuthology.orchestra.run.gibba020.stdout:mon.c: {} 2021-03-14T06:23:23.084 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:37.782 INFO:teuthology.orchestra.run.gibba020.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2021-03-13T17:24:36.003551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1336279040,"num_objects":1175,"num_object_clones":168,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":33,"num_read":2580,"num_read_kb":280699,"num_write":4572,"num_write_kb":1342608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3286,"num_bytes_recovered":3858341888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6997,"ondisk_log_size":6997,"up":86,"acting":86,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2333940,"kb_used_data":2116724,"kb_used_omap":37,"kb_used_meta":217114,"kb_avail":752640780,"statfs":{"total":773094113280,"available":770704158720,"internally_reserved":0,"allocated":2167525376,"data_stored":2258364495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":37912,"internal_metadata":222325736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":319488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160492"},"pg_stats":[{"pgid":"3.f","version":"159'352","reported_seq":"823","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.307101+0000","last_change":"2021-03-13T17:24:19.085412+0000","last_active":"2021-03-13T17:24:32.307101+0000","last_peered":"2021-03-13T17:24:32.307101+0000","last_clean":"2021-03-13T17:24:32.307101+0000","last_became_active":"2021-03-13T17:24:19.084875+0000","last_became_peered":"2021-03-13T17:24:19.084875+0000","last_unstale":"2021-03-13T17:24:32.307101+0000","last_undegraded":"2021-03-13T17:24:32.307101+0000","last_fullsized":"2021-03-13T17:24:32.307101+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'352","last_scrub_stamp":"2021-03-13T17:24:14.287154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:14.287154+0000","log_size":352,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78168064,"num_objects":60,"num_object_clones":5,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":134,"num_read_kb":17152,"num_write":234,"num_write_kb":76336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":303,"num_bytes_recovered":401883136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"163'453","reported_seq":"1279","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846761+0000","last_change":"2021-03-13T17:24:30.951410+0000","last_active":"2021-03-13T17:24:31.846761+0000","last_peered":"2021-03-13T17:24:31.846761+0000","last_clean":"2021-03-13T17:24:31.846761+0000","last_became_active":"2021-03-13T17:24:20.085873+0000","last_became_peered":"2021-03-13T17:24:20.085873+0000","last_unstale":"2021-03-13T17:24:31.846761+0000","last_undegraded":"2021-03-13T17:24:31.846761+0000","last_fullsized":"2021-03-13T17:24:31.846761+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'453","last_scrub_stamp":"2021-03-13T17:24:30.951350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:30.951350+0000","log_size":453,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":88023040,"num_objects":75,"num_object_clones":8,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":0,"num_read":163,"num_read_kb":9614,"num_write":352,"num_write_kb":88832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":279969792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"163'577","reported_seq":"1330","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221911+0000","last_change":"2021-03-13T17:24:19.086770+0000","last_active":"2021-03-13T17:24:32.221911+0000","last_peered":"2021-03-13T17:24:32.221911+0000","last_clean":"2021-03-13T17:24:32.221911+0000","last_became_active":"2021-03-13T17:24:19.086193+0000","last_became_peered":"2021-03-13T17:24:19.086193+0000","last_unstale":"2021-03-13T17:24:32.221911+0000","last_undegraded":"2021-03-13T17:24:32.221911+0000","last_fullsized":"2021-03-13T17:24:32.221911+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'549","last_scrub_stamp":"2021-03-13T17:23:42.168126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:23:42.168126+0000","log_size":577,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82747392,"num_objects":78,"num_object_clones":15,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":5,"num_read":151,"num_read_kb":7849,"num_write":320,"num_write_kb":82952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":171270144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"159'366","reported_seq":"856","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.307087+0000","last_change":"2021-03-13T17:24:13.335321+0000","last_active":"2021-03-13T17:24:32.307087+0000","last_peered":"2021-03-13T17:24:32.307087+0000","last_clean":"2021-03-13T17:24:32.307087+0000","last_became_active":"2021-03-13T17:23:44.820434+0000","last_became_peered":"2021-03-13T17:23:44.820434+0000","last_unstale":"2021-03-13T17:24:32.307087+0000","last_undegraded":"2021-03-13T17:24:32.307087+0000","last_fullsized":"2021-03-13T17:24:32.307087+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'366","last_scrub_stamp":"2021-03-13T17:24:13.335268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:13.335268+0000","log_size":366,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":82149376,"num_objects":73,"num_object_clones":13,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":3,"num_read":159,"num_read_kb":19752,"num_write":282,"num_write_kb":82280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"212","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846146+0000","last_change":"2021-03-13T17:24:16.079869+0000","last_active":"2021-03-13T17:24:31.846146+0000","last_peered":"2021-03-13T17:24:31.846146+0000","last_clean":"2021-03-13T17:24:31.846146+0000","last_became_active":"2021-03-13T17:24:16.079629+0000","last_became_peered":"2021-03-13T17:24:16.079629+0000","last_unstale":"2021-03-13T17:24:31.846146+0000","last_undegraded":"2021-03-13T17:24:31.846146+0000","last_fullsized":"2021-03-13T17:24:31.846146+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:00.011602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:23:00.011602+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"163'396","reported_seq":"1137","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.400798+0000","last_change":"2021-03-13T17:24:14.587952+0000","last_active":"2021-03-13T17:24:32.400798+0000","last_peered":"2021-03-13T17:24:32.400798+0000","last_clean":"2021-03-13T17:24:32.400798+0000","last_became_active":"2021-03-13T17:23:51.844016+0000","last_became_peered":"2021-03-13T17:23:51.844016+0000","last_unstale":"2021-03-13T17:24:32.400798+0000","last_undegraded":"2021-03-13T17:24:32.400798+0000","last_fullsized":"2021-03-13T17:24:32.400798+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'396","last_scrub_stamp":"2021-03-13T17:24:14.587878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:14.587878+0000","log_size":396,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67182592,"num_objects":59,"num_object_clones":9,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":136,"num_read_kb":16143,"num_write":240,"num_write_kb":69528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":140566528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"104","reported_epoch":"193","state":"active+clean","last_fresh":"2021-03-13T17:24:34.178734+0000","last_change":"2021-03-13T17:24:34.178734+0000","last_active":"2021-03-13T17:24:34.178734+0000","last_peered":"2021-03-13T17:24:34.178734+0000","last_clean":"2021-03-13T17:24:34.178734+0000","last_became_active":"2021-03-13T17:24:20.085850+0000","last_became_peered":"2021-03-13T17:24:20.085850+0000","last_unstale":"2021-03-13T17:24:34.178734+0000","last_undegraded":"2021-03-13T17:24:34.178734+0000","last_fullsized":"2021-03-13T17:24:34.178734+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:34.178683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:34.178683+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"168'531","reported_seq":"1576","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221804+0000","last_change":"2021-03-13T17:24:11.106354+0000","last_active":"2021-03-13T17:24:32.221804+0000","last_peered":"2021-03-13T17:24:32.221804+0000","last_clean":"2021-03-13T17:24:32.221804+0000","last_became_active":"2021-03-13T17:24:00.826627+0000","last_became_peered":"2021-03-13T17:24:00.826627+0000","last_unstale":"2021-03-13T17:24:32.221804+0000","last_undegraded":"2021-03-13T17:24:32.221804+0000","last_fullsized":"2021-03-13T17:24:32.221804+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'531","last_scrub_stamp":"2021-03-13T17:24:11.106285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:11.106285+0000","log_size":531,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":97943552,"num_objects":87,"num_object_clones":16,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":182,"num_read_kb":25596,"num_write":332,"num_write_kb":97928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":288677888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"159'388","reported_seq":"1662","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221816+0000","last_change":"2021-03-13T17:24:25.826211+0000","last_active":"2021-03-13T17:24:32.221816+0000","last_peered":"2021-03-13T17:24:32.221816+0000","last_clean":"2021-03-13T17:24:32.221816+0000","last_became_active":"2021-03-13T17:24:25.825682+0000","last_became_peered":"2021-03-13T17:24:25.825682+0000","last_unstale":"2021-03-13T17:24:32.221816+0000","last_undegraded":"2021-03-13T17:24:32.221816+0000","last_fullsized":"2021-03-13T17:24:32.221816+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'388","last_scrub_stamp":"2021-03-13T17:24:03.001731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:03.001731+0000","log_size":388,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71655424,"num_objects":60,"num_object_clones":6,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":1,"num_read":140,"num_read_kb":13338,"num_write":214,"num_write_kb":72384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":359882752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"211","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846145+0000","last_change":"2021-03-13T17:24:02.211870+0000","last_active":"2021-03-13T17:24:31.846145+0000","last_peered":"2021-03-13T17:24:31.846145+0000","last_clean":"2021-03-13T17:24:31.846145+0000","last_became_active":"2021-03-13T17:22:30.218106+0000","last_became_peered":"2021-03-13T17:22:30.218106+0000","last_unstale":"2021-03-13T17:24:31.846145+0000","last_undegraded":"2021-03-13T17:24:31.846145+0000","last_fullsized":"2021-03-13T17:24:31.846145+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:02.211823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:02.211823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"166'451","reported_seq":"1354","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850202+0000","last_change":"2021-03-13T17:24:28.837664+0000","last_active":"2021-03-13T17:24:31.850202+0000","last_peered":"2021-03-13T17:24:31.850202+0000","last_clean":"2021-03-13T17:24:31.850202+0000","last_became_active":"2021-03-13T17:24:28.837056+0000","last_became_peered":"2021-03-13T17:24:28.837056+0000","last_unstale":"2021-03-13T17:24:31.850202+0000","last_undegraded":"2021-03-13T17:24:31.850202+0000","last_fullsized":"2021-03-13T17:24:31.850202+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"166'451","last_scrub_stamp":"2021-03-13T17:24:11.260440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:11.260440+0000","log_size":451,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":106553344,"num_objects":89,"num_object_clones":14,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":4,"num_read":213,"num_read_kb":29856,"num_write":337,"num_write_kb":107360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":220209152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"209","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846259+0000","last_change":"2021-03-13T17:24:12.167531+0000","last_active":"2021-03-13T17:24:31.846259+0000","last_peered":"2021-03-13T17:24:31.846259+0000","last_clean":"2021-03-13T17:24:31.846259+0000","last_became_active":"2021-03-13T17:22:30.218176+0000","last_became_peered":"2021-03-13T17:22:30.218176+0000","last_unstale":"2021-03-13T17:24:31.846259+0000","last_undegraded":"2021-03-13T17:24:31.846259+0000","last_fullsized":"2021-03-13T17:24:31.846259+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:12.167470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:12.167470+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"198","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.793642+0000","last_change":"2021-03-13T17:24:11.832836+0000","last_active":"2021-03-13T17:24:32.793642+0000","last_peered":"2021-03-13T17:24:32.793642+0000","last_clean":"2021-03-13T17:24:32.793642+0000","last_became_active":"2021-03-13T17:21:16.889215+0000","last_became_peered":"2021-03-13T17:21:16.889215+0000","last_unstale":"2021-03-13T17:24:32.793642+0000","last_undegraded":"2021-03-13T17:24:32.793642+0000","last_fullsized":"2021-03-13T17:24:32.793642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:11.832787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:24:11.832787+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"163'402","reported_seq":"355","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.189704+0000","last_change":"2021-03-13T17:24:32.189667+0000","last_active":"2021-03-13T17:24:32.189704+0000","last_peered":"2021-03-13T17:24:32.189704+0000","last_clean":"2021-03-13T17:24:32.189704+0000","last_became_active":"2021-03-13T17:24:22.089771+0000","last_became_peered":"2021-03-13T17:24:22.089771+0000","last_unstale":"2021-03-13T17:24:32.189704+0000","last_undegraded":"2021-03-13T17:24:32.189704+0000","last_fullsized":"2021-03-13T17:24:32.189704+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'402","last_scrub_stamp":"2021-03-13T17:24:32.189586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:32.189586+0000","log_size":402,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81076224,"num_objects":69,"num_object_clones":8,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":0,"num_read":159,"num_read_kb":17064,"num_write":298,"num_write_kb":80664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":281,"num_bytes_recovered":347865088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"137","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221795+0000","last_change":"2021-03-13T17:24:31.178985+0000","last_active":"2021-03-13T17:24:32.221795+0000","last_peered":"2021-03-13T17:24:32.221795+0000","last_clean":"2021-03-13T17:24:32.221795+0000","last_became_active":"2021-03-13T17:22:30.218287+0000","last_became_peered":"2021-03-13T17:22:30.218287+0000","last_unstale":"2021-03-13T17:24:32.221795+0000","last_undegraded":"2021-03-13T17:24:32.221795+0000","last_fullsized":"2021-03-13T17:24:32.221795+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:31.178933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:31.178933+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"189","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221822+0000","last_change":"2021-03-13T17:23:40.126429+0000","last_active":"2021-03-13T17:24:32.221822+0000","last_peered":"2021-03-13T17:24:32.221822+0000","last_clean":"2021-03-13T17:24:32.221822+0000","last_became_active":"2021-03-13T17:21:16.889159+0000","last_became_peered":"2021-03-13T17:21:16.889159+0000","last_unstale":"2021-03-13T17:24:32.221822+0000","last_undegraded":"2021-03-13T17:24:32.221822+0000","last_fullsized":"2021-03-13T17:24:32.221822+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:40.126354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:23:40.126354+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"163'371","reported_seq":"1229","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.307061+0000","last_change":"2021-03-13T17:24:20.086758+0000","last_active":"2021-03-13T17:24:32.307061+0000","last_peered":"2021-03-13T17:24:32.307061+0000","last_clean":"2021-03-13T17:24:32.307061+0000","last_became_active":"2021-03-13T17:24:20.086099+0000","last_became_peered":"2021-03-13T17:24:20.086099+0000","last_unstale":"2021-03-13T17:24:32.307061+0000","last_undegraded":"2021-03-13T17:24:32.307061+0000","last_fullsized":"2021-03-13T17:24:32.307061+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'371","last_scrub_stamp":"2021-03-13T17:24:05.333570+0000","last_deep_scrub":"163'371","last_deep_scrub_stamp":"2021-03-13T17:24:05.333570+0000","last_clean_scrub_stamp":"2021-03-13T17:24:05.333570+0000","log_size":371,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":77111296,"num_objects":70,"num_object_clones":8,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":2,"num_read":156,"num_read_kb":10755,"num_write":252,"num_write_kb":79592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":157777920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"106","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.306986+0000","last_change":"2021-03-13T17:24:16.078943+0000","last_active":"2021-03-13T17:24:32.306986+0000","last_peered":"2021-03-13T17:24:32.306986+0000","last_clean":"2021-03-13T17:24:32.306986+0000","last_became_active":"2021-03-13T17:24:16.078723+0000","last_became_peered":"2021-03-13T17:24:16.078723+0000","last_unstale":"2021-03-13T17:24:32.306986+0000","last_undegraded":"2021-03-13T17:24:32.306986+0000","last_fullsized":"2021-03-13T17:24:32.306986+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:41.970823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:22:51.764891+0000","last_clean_scrub_stamp":"2021-03-13T17:23:41.970823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"193","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.401008+0000","last_change":"2021-03-13T17:24:29.478854+0000","last_active":"2021-03-13T17:24:32.401008+0000","last_peered":"2021-03-13T17:24:32.401008+0000","last_clean":"2021-03-13T17:24:32.401008+0000","last_became_active":"2021-03-13T17:21:16.826066+0000","last_became_peered":"2021-03-13T17:21:16.826066+0000","last_unstale":"2021-03-13T17:24:32.401008+0000","last_undegraded":"2021-03-13T17:24:32.401008+0000","last_fullsized":"2021-03-13T17:24:32.401008+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:29.478743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:24:29.478743+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"165'432","reported_seq":"1554","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.793574+0000","last_change":"2021-03-13T17:24:30.992014+0000","last_active":"2021-03-13T17:24:32.793574+0000","last_peered":"2021-03-13T17:24:32.793574+0000","last_clean":"2021-03-13T17:24:32.793574+0000","last_became_active":"2021-03-13T17:24:24.825030+0000","last_became_peered":"2021-03-13T17:24:24.825030+0000","last_unstale":"2021-03-13T17:24:32.793574+0000","last_undegraded":"2021-03-13T17:24:32.793574+0000","last_fullsized":"2021-03-13T17:24:32.793574+0000","mapping_epoch":184,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":185,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'432","last_scrub_stamp":"2021-03-13T17:24:30.991958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:30.991958+0000","log_size":432,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":79945728,"num_objects":75,"num_object_clones":11,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":6,"num_read":150,"num_read_kb":10247,"num_write":238,"num_write_kb":79800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":337412096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"206","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221800+0000","last_change":"2021-03-13T17:24:17.079930+0000","last_active":"2021-03-13T17:24:32.221800+0000","last_peered":"2021-03-13T17:24:32.221800+0000","last_clean":"2021-03-13T17:24:32.221800+0000","last_became_active":"2021-03-13T17:24:17.079684+0000","last_became_peered":"2021-03-13T17:24:17.079684+0000","last_unstale":"2021-03-13T17:24:32.221800+0000","last_undegraded":"2021-03-13T17:24:32.221800+0000","last_fullsized":"2021-03-13T17:24:32.221800+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:03.181905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:23:03.181905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"202","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846161+0000","last_change":"2021-03-13T17:24:03.173703+0000","last_active":"2021-03-13T17:24:31.846161+0000","last_peered":"2021-03-13T17:24:31.846161+0000","last_clean":"2021-03-13T17:24:31.846161+0000","last_became_active":"2021-03-13T17:22:27.815701+0000","last_became_peered":"2021-03-13T17:22:27.815701+0000","last_unstale":"2021-03-13T17:24:31.846161+0000","last_undegraded":"2021-03-13T17:24:31.846161+0000","last_fullsized":"2021-03-13T17:24:31.846161+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:03.173581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:03.173581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"163'443","reported_seq":"19","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850252+0000","last_change":"2021-03-13T17:24:27.833596+0000","last_active":"2021-03-13T17:24:31.850252+0000","last_peered":"2021-03-13T17:24:31.850252+0000","last_clean":"2021-03-13T17:24:31.850252+0000","last_became_active":"2021-03-13T17:24:27.833068+0000","last_became_peered":"2021-03-13T17:24:27.833068+0000","last_unstale":"2021-03-13T17:24:31.850252+0000","last_undegraded":"2021-03-13T17:24:31.850252+0000","last_fullsized":"2021-03-13T17:24:31.850252+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'290","last_scrub_stamp":"2021-03-13T17:23:01.007287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:23:01.007287+0000","log_size":443,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":86474752,"num_objects":80,"num_object_clones":15,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":4,"num_read":167,"num_read_kb":17697,"num_write":285,"num_write_kb":85184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":237,"num_bytes_recovered":274702336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"26","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850247+0000","last_change":"2021-03-13T17:24:16.078397+0000","last_active":"2021-03-13T17:24:31.850247+0000","last_peered":"2021-03-13T17:24:31.850247+0000","last_clean":"2021-03-13T17:24:31.850247+0000","last_became_active":"2021-03-13T17:24:16.078149+0000","last_became_peered":"2021-03-13T17:24:16.078149+0000","last_unstale":"2021-03-13T17:24:31.850247+0000","last_undegraded":"2021-03-13T17:24:31.850247+0000","last_fullsized":"2021-03-13T17:24:31.850247+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:04.470273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:04.470273+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"194","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221794+0000","last_change":"2021-03-13T17:24:29.243316+0000","last_active":"2021-03-13T17:24:32.221794+0000","last_peered":"2021-03-13T17:24:32.221794+0000","last_clean":"2021-03-13T17:24:32.221794+0000","last_became_active":"2021-03-13T17:21:16.825340+0000","last_became_peered":"2021-03-13T17:21:16.825340+0000","last_unstale":"2021-03-13T17:24:32.221794+0000","last_undegraded":"2021-03-13T17:24:32.221794+0000","last_fullsized":"2021-03-13T17:24:32.221794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:29.243265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:24:29.243265+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"163'356","reported_seq":"1575","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.852825+0000","last_change":"2021-03-13T17:24:31.851798+0000","last_active":"2021-03-13T17:24:31.852825+0000","last_peered":"2021-03-13T17:24:31.852825+0000","last_clean":"2021-03-13T17:24:31.852825+0000","last_became_active":"2021-03-13T17:24:31.851303+0000","last_became_peered":"2021-03-13T17:24:31.851303+0000","last_unstale":"2021-03-13T17:24:31.852825+0000","last_undegraded":"2021-03-13T17:24:31.852825+0000","last_fullsized":"2021-03-13T17:24:31.852825+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'356","last_scrub_stamp":"2021-03-13T17:24:07.156607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:07.156607+0000","log_size":356,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":89178112,"num_objects":84,"num_object_clones":8,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":3,"num_read":171,"num_read_kb":3084,"num_write":294,"num_write_kb":89344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":314007552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"130","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221857+0000","last_change":"2021-03-13T17:23:02.167242+0000","last_active":"2021-03-13T17:24:32.221857+0000","last_peered":"2021-03-13T17:24:32.221857+0000","last_clean":"2021-03-13T17:24:32.221857+0000","last_became_active":"2021-03-13T17:22:30.218198+0000","last_became_peered":"2021-03-13T17:22:30.218198+0000","last_unstale":"2021-03-13T17:24:32.221857+0000","last_undegraded":"2021-03-13T17:24:32.221857+0000","last_fullsized":"2021-03-13T17:24:32.221857+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:02.167166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:23:02.167166+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"205","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221817+0000","last_change":"2021-03-13T17:24:16.079288+0000","last_active":"2021-03-13T17:24:32.221817+0000","last_peered":"2021-03-13T17:24:32.221817+0000","last_clean":"2021-03-13T17:24:32.221817+0000","last_became_active":"2021-03-13T17:24:16.078921+0000","last_became_peered":"2021-03-13T17:24:16.078921+0000","last_unstale":"2021-03-13T17:24:32.221817+0000","last_undegraded":"2021-03-13T17:24:32.221817+0000","last_fullsized":"2021-03-13T17:24:32.221817+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:17.280027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:23:17.280027+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"163'529","reported_seq":"1306","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.848630+0000","last_change":"2021-03-13T17:24:19.085934+0000","last_active":"2021-03-13T17:24:31.848630+0000","last_peered":"2021-03-13T17:24:31.848630+0000","last_clean":"2021-03-13T17:24:31.848630+0000","last_became_active":"2021-03-13T17:24:19.084837+0000","last_became_peered":"2021-03-13T17:24:19.084837+0000","last_unstale":"2021-03-13T17:24:31.848630+0000","last_undegraded":"2021-03-13T17:24:31.848630+0000","last_fullsized":"2021-03-13T17:24:31.848630+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'529","last_scrub_stamp":"2021-03-13T17:24:03.139074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:03.139074+0000","log_size":529,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90554368,"num_objects":71,"num_object_clones":10,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":163,"num_read_kb":29498,"num_write":360,"num_write_kb":92688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":186032128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"21","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.846751+0000","last_change":"2021-03-13T17:24:21.086798+0000","last_active":"2021-03-13T17:24:31.846751+0000","last_peered":"2021-03-13T17:24:31.846751+0000","last_clean":"2021-03-13T17:24:31.846751+0000","last_became_active":"2021-03-13T17:24:21.086606+0000","last_became_peered":"2021-03-13T17:24:21.086606+0000","last_unstale":"2021-03-13T17:24:31.846751+0000","last_undegraded":"2021-03-13T17:24:31.846751+0000","last_fullsized":"2021-03-13T17:24:31.846751+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:18.253463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:23:18.253463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"159'440","reported_seq":"913","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850312+0000","last_change":"2021-03-13T17:24:19.085588+0000","last_active":"2021-03-13T17:24:31.850312+0000","last_peered":"2021-03-13T17:24:31.850312+0000","last_clean":"2021-03-13T17:24:31.850312+0000","last_became_active":"2021-03-13T17:24:19.085021+0000","last_became_peered":"2021-03-13T17:24:19.085021+0000","last_unstale":"2021-03-13T17:24:31.850312+0000","last_undegraded":"2021-03-13T17:24:31.850312+0000","last_fullsized":"2021-03-13T17:24:31.850312+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'440","last_scrub_stamp":"2021-03-13T17:24:13.119795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:13.119795+0000","log_size":440,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":73957376,"num_objects":70,"num_object_clones":7,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":3,"num_read":167,"num_read_kb":22160,"num_write":271,"num_write_kb":73816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"26","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850297+0000","last_change":"2021-03-13T17:24:16.076990+0000","last_active":"2021-03-13T17:24:31.850297+0000","last_peered":"2021-03-13T17:24:31.850297+0000","last_clean":"2021-03-13T17:24:31.850297+0000","last_became_active":"2021-03-13T17:24:16.076803+0000","last_became_peered":"2021-03-13T17:24:16.076803+0000","last_unstale":"2021-03-13T17:24:31.850297+0000","last_undegraded":"2021-03-13T17:24:31.850297+0000","last_fullsized":"2021-03-13T17:24:31.850297+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:23:00.689527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:15.811834+0000","last_clean_scrub_stamp":"2021-03-13T17:23:00.689527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"132","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.400791+0000","last_change":"2021-03-13T17:24:30.228219+0000","last_active":"2021-03-13T17:24:32.400791+0000","last_peered":"2021-03-13T17:24:32.400791+0000","last_clean":"2021-03-13T17:24:32.400791+0000","last_became_active":"2021-03-13T17:24:30.227966+0000","last_became_peered":"2021-03-13T17:24:30.227966+0000","last_unstale":"2021-03-13T17:24:32.400791+0000","last_undegraded":"2021-03-13T17:24:32.400791+0000","last_fullsized":"2021-03-13T17:24:32.400791+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:16.292953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:14.808330+0000","last_clean_scrub_stamp":"2021-03-13T17:24:16.292953+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":"207","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:32.221857+0000","last_change":"2021-03-13T17:24:28.247617+0000","last_active":"2021-03-13T17:24:32.221857+0000","last_peered":"2021-03-13T17:24:32.221857+0000","last_clean":"2021-03-13T17:24:32.221857+0000","last_became_active":"2021-03-13T17:23:41.861826+0000","last_became_peered":"2021-03-13T17:23:41.861826+0000","last_unstale":"2021-03-13T17:24:32.221857+0000","last_undegraded":"2021-03-13T17:24:32.221857+0000","last_fullsized":"2021-03-13T17:24:32.221857+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T17:24:28.247507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:22:26.974069+0000","last_clean_scrub_stamp":"2021-03-13T17:24:28.247507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"163'510","reported_seq":"1661","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T17:24:31.850383+0000","last_change":"2021-03-13T17:24:19.085843+0000","last_active":"2021-03-13T17:24:31.850383+0000","last_peered":"2021-03-13T17:24:31.850383+0000","last_clean":"2021-03-13T17:24:31.850383+0000","last_became_active":"2021-03-13T17:24:19.085346+0000","last_became_peered":"2021-03-13T17:24:19.085346+0000","last_unstale":"2021-03-13T17:24:31.850383+0000","last_undegraded":"2021-03-13T17:24:31.850383+0000","last_fullsized":"2021-03-13T17:24:31.850383+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"163'510","last_scrub_stamp":"2021-03-13T17:24:09.211622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T17:21:29.045275+0000","last_clean_scrub_stamp":"2021-03-13T17:24:09.211622+0000","log_size":510,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83558400,"num_objects":75,"num_object_clones":15,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":2,"num_read":169,"num_read_kb":30894,"num_write":263,"num_write_kb":83920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":378085376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1336279040,"num_objects":1175,"num_object_clones":168,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":33,"num_read":2580,"num_read_kb":280699,"num_write":4572,"num_write_kb":1342608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3286,"num_bytes_recovered":3858341888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2135941120,"data_stored":2235482112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6997,"ondisk_log_size":6997,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542188,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":315892,"kb_used_data":283628,"kb_used_omap":0,"kb_used_meta":32256,"kb_avail":94055948,"statfs":{"total":96636764160,"available":96313290752,"internally_reserved":0,"allocated":290435072,"data_stored":305341825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33030144},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:15 2021","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":1.4039999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":1.2669999999999999}]},{"osd":1,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.73099999999999998,"15min":0.70999999999999996},"min":{"1min":0.371,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":1.492},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77700000000000002,"15min":0.755},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":1.323}]},{"osd":2,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.80600000000000005,"15min":0.78800000000000003},"min":{"1min":0.433,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.3779999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.80300000000000005,"15min":0.78300000000000003},"min":{"1min":0.57699999999999996,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.3100000000000001}]},{"osd":3,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.83899999999999997,"15min":0.82899999999999996},"min":{"1min":0.55600000000000005,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.82799999999999996,"15min":0.79400000000000004},"min":{"1min":0.58399999999999996,"5min":0.502,"15min":0.502},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":1.1859999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":5,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.90900000000000003,"15min":0.93799999999999994},"min":{"1min":0.46200000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.111,"5min":1.278,"15min":1.278},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.84399999999999997,"15min":0.85399999999999998},"min":{"1min":0.501,"5min":0.497,"15min":0.497},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.90400000000000003,"15min":0.91300000000000003},"min":{"1min":0.54500000000000004,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.226,"5min":1.296,"15min":1.296},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.995,"5min":0.97999999999999998,"15min":0.99199999999999999},"min":{"1min":0.63800000000000001,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.29,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.96399999999999997}]}]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":198444,"kb_used_data":158948,"kb_used_omap":0,"kb_used_meta":39488,"kb_avail":94173396,"statfs":{"total":96636764160,"available":96433557504,"internally_reserved":0,"allocated":162762752,"data_stored":178161025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40435712},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:13 2021","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.97399999999999998}]},{"osd":1,"last update":"Sat Mar 13 17:24:16 2021","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.82499999999999996,"15min":0.82799999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1899999999999999,"5min":1.335,"15min":1.335},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":1.2969999999999999}]},{"osd":2,"last update":"Sat Mar 13 17:24:16 2021","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.82799999999999996,"15min":0.82199999999999995},"min":{"1min":0.495,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.175,"5min":1.446,"15min":1.446},"last":1.353},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85999999999999999,"15min":0.86099999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1439999999999999,"5min":1.319,"15min":1.319},"last":0.98299999999999998}]},{"osd":3,"last update":"Sat Mar 13 17:24:16 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.89000000000000001,"15min":0.88800000000000001},"min":{"1min":0.63700000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.262,"5min":1.319,"15min":1.319},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.91700000000000004},"min":{"1min":0.51200000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.131,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":1.2210000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":5,"last update":"Sat Mar 13 17:24:16 2021","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.78400000000000003,"15min":0.78200000000000003},"min":{"1min":0.45100000000000001,"5min":0.435,"15min":0.435},"max":{"1min":1.0369999999999999,"5min":1.325,"15min":1.325},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.86099999999999999,"15min":0.871},"min":{"1min":0.49199999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0049999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Sat Mar 13 17:24:16 2021","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.85199999999999998,"15min":0.83699999999999997},"min":{"1min":0.59099999999999997,"5min":0.501,"15min":0.501},"max":{"1min":1.2430000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.91400000000000003,"15min":0.91100000000000003},"min":{"1min":0.54600000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2310000000000001,"5min":1.647,"15min":1.647},"last":1.2849999999999999}]}]},{"osd":0,"up_from":148,"seq":635655159821,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":268552,"kb_used_data":254720,"kb_used_omap":36,"kb_used_meta":13787,"kb_avail":94103288,"statfs":{"total":96636764160,"available":96361766912,"internally_reserved":0,"allocated":260833280,"data_stored":267257217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":36940,"internal_metadata":14118836},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574893,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160188,"kb_used_data":128500,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":94211652,"statfs":{"total":96636764160,"available":96472731648,"internally_reserved":0,"allocated":131584000,"data_stored":144843208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:19 2021","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":0.79200000000000004}]},{"osd":1,"last update":"Sat Mar 13 17:24:24 2021","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.76400000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.2030000000000001,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.03},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.85599999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2430000000000001,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.69699999999999995}]},{"osd":2,"last update":"Sat Mar 13 17:24:24 2021","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.80100000000000005,"15min":0.80200000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.77600000000000002,"15min":0.78000000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.095,"5min":1.234,"15min":1.234},"last":0.71199999999999997}]},{"osd":3,"last update":"Sat Mar 13 17:24:24 2021","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.86399999999999999,"15min":0.878},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.3500000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.93600000000000005,"15min":0.96799999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.177,"5min":1.403,"15min":1.403},"last":0.72599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]},{"osd":6,"last update":"Sat Mar 13 17:24:24 2021","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.75600000000000001,"15min":0.77400000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.84499999999999997,"15min":0.85299999999999998},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0820000000000001,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.90700000000000003}]},{"osd":7,"last update":"Sat Mar 13 17:24:24 2021","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.86499999999999999,"15min":0.877},"min":{"1min":0.54800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.879,"15min":0.89300000000000002},"min":{"1min":0.59299999999999997,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0109999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.99099999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":277448,"kb_used_data":250816,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94094392,"statfs":{"total":96636764160,"available":96352657408,"internally_reserved":0,"allocated":256835584,"data_stored":282826113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:17 2021","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Mar 13 17:24:22 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54100000000000004,"15min":0.54200000000000004},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.88900000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.748}]},{"osd":3,"last update":"Sat Mar 13 17:24:22 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67600000000000005,"15min":0.69999999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.64800000000000002,"15min":0.65300000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.92300000000000004,"5min":0.999,"15min":0.999},"last":0.73399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.411}]},{"osd":5,"last update":"Sat Mar 13 17:24:22 2021","interfaces":[{"interface":"back","average":{"1min":1,"5min":1.0369999999999999,"15min":1.0589999999999999},"min":{"1min":0.52200000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.444,"5min":1.542,"15min":1.542},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.95899999999999996,"5min":1.0609999999999999,"15min":1.093},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4750000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"last":1.4299999999999999}]},{"osd":6,"last update":"Sat Mar 13 17:24:28 2021","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1200000000000001,"15min":1.139},"min":{"1min":0.67600000000000005,"5min":0.505,"15min":0.505},"max":{"1min":1.587,"5min":1.6910000000000001,"15min":1.6910000000000001},"last":1.552},{"interface":"front","average":{"1min":1.109,"5min":1.161,"15min":1.1930000000000001},"min":{"1min":0.71599999999999997,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6359999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.621}]},{"osd":7,"last update":"Sat Mar 13 17:24:22 2021","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.123,"15min":1.141},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.528,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.089,"5min":1.105,"15min":1.1200000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.4750000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":394680,"kb_used_data":368048,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":93977160,"statfs":{"total":96636764160,"available":96232611840,"internally_reserved":0,"allocated":376881152,"data_stored":375793025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:18 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.7509999999999999,"5min":3.7509999999999999,"15min":3.7509999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Sat Mar 13 17:24:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.59599999999999997,"15min":0.621},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83599999999999997,"5min":1.488,"15min":1.488},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.71299999999999997,"15min":0.76600000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.92400000000000004,"5min":3.585,"15min":3.585},"last":0.41199999999999998}]},{"osd":3,"last update":"Sat Mar 13 17:24:23 2021","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.62,"15min":0.63600000000000001},"min":{"1min":0.39900000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77600000000000002,"5min":1.038,"15min":1.038},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.133,"5min":1.6599999999999999,"15min":1.6599999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.448}]},{"osd":5,"last update":"Sat Mar 13 17:24:23 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.95299999999999996,"15min":0.97799999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.597,"5min":1.597,"15min":1.597},"last":1.1859999999999999},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.98899999999999999,"15min":1.026},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.518,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.2270000000000001}]},{"osd":6,"last update":"Sat Mar 13 17:24:23 2021","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":1.087,"15min":1.133},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.4410000000000001,"5min":2.4449999999999998,"15min":2.4449999999999998},"last":1.177},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0669999999999999,"15min":1.0760000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.2490000000000001}]},{"osd":7,"last update":"Sat Mar 13 17:24:23 2021","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":1.0740000000000001,"15min":1.131},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.306,"5min":3.3220000000000001,"15min":3.3220000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.077,"15min":1.109},"min":{"1min":0.64400000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.504,"5min":2.742,"15min":2.742},"last":1.3049999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542189,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":193492,"kb_used_data":165452,"kb_used_omap":0,"kb_used_meta":28032,"kb_avail":94178348,"statfs":{"total":96636764160,"available":96438628352,"internally_reserved":0,"allocated":169422848,"data_stored":186459521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28704768},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 17:22:16 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.71799999999999997}]},{"osd":1,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.56499999999999995,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.66700000000000004,"15min":0.67100000000000004},"min":{"1min":0.33900000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.78200000000000003,"5min":1.27,"15min":1.27},"last":0.498},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.62,"15min":0.627},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.89600000000000002,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.54800000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.26}]},{"osd":5,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94499999999999995,"15min":0.94299999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.3380000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.01,"15min":1.02},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.25,"5min":1.4850000000000001,"15min":1.4850000000000001},"last":0.78700000000000003}]},{"osd":6,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.94799999999999995},"min":{"1min":0.48399999999999999,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.343,"5min":1.542,"15min":1.542},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":1.024,"15min":1.004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.389,"5min":1.619,"15min":1.619},"last":1.4139999999999999}]},{"osd":7,"last update":"Sat Mar 13 17:24:19 2021","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.97699999999999998,"15min":0.98299999999999998},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.2929999999999999,"5min":1.7010000000000001,"15min":1.7010000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.002,"15min":1.002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2330000000000001,"5min":1.6679999999999999,"15min":1.6679999999999999},"last":1.0209999999999999}]}]},{"osd":4,"up_from":142,"seq":609885356046,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":525244,"kb_used_data":506612,"kb_used_omap":0,"kb_used_meta":18623,"kb_avail":93846596,"statfs":{"total":96636764160,"available":96098914304,"internally_reserved":0,"allocated":518770688,"data_stored":517682561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":19070004},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.363}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0640000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":258506752,"data_stored":266018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254509056,"data_stored":281587712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":374554624,"data_stored":374554624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":167096320,"data_stored":185221120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":516444160,"data_stored":516444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":129261568,"data_stored":143605760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":160436224,"data_stored":176922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":275132416,"data_stored":291127296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2021-03-13T17:24:37.785 INFO:tasks.ceph.ceph_manager.ceph:recovered! , 2021-03-13T17:24:37.786 DEBUG:teuthology.orchestra.run.gibba020:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2021-03-14T06:23:23.087 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:24:42.808 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-03-13T17:23:02.167166+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=23, tm_sec=2, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=24, tm_sec=38, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T17:24:42.809 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2021-03-13T17:24:36.134249+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=24, tm_sec=36, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=24, tm_sec=38, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T17:24:42.809 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2021-03-13T17:24:03.139074+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=24, tm_sec=3, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=17, tm_min=24, tm_sec=38, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:23.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:25:24.117 INFO:tasks.ceph:Checking cluster log for badness... , 2021-03-13T17:25:24.118 DEBUG:teuthology.orchestra.run.gibba020:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | head -n 1 , 2021-03-13T17:25:24.171 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@gibba020.front.sepia.ceph.com 2021-03-14T06:23:23.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T17:25:56.186 INFO:teuthology.orchestra.run.gibba020.stdout:^* hv04.front.sepia.ceph.com 3 6 377 52 -815us[ -814us] +/- 70ms , 2021-03-13T17:25:56.186 DEBUG:teuthology.orchestra.run.gibba028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , bash: ntpq: command not found 2021-03-14T06:23:23.089 WARNING:scrape:Ignoring malparsed backtrace: running.get() , File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2021-03-14T06:23:24.172 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f74ecd61308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f74ecd61522] 2021-03-14T06:23:24.173 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.844 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 0.74994940499999996, , 2021-03-13T15:17:20.844 INFO:teuthology.orchestra.run.gibba007.stdout: "type_data": { , 2021-03-13T15:17:20.844 INFO:teuthology.orchestra.run.gibba007.stdout: "flag_point": "started", 2021-03-14T06:23:24.174 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.876 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "waiting for blocked object", , 2021-03-13T15:17:20.876 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:13:38.728828+0000", , 2021-03-13T15:17:20.876 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 0.0040548939999999999 2021-03-14T06:23:24.176 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.899 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "dispatched", , 2021-03-13T15:17:20.900 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:13:38.728834+0000", , 2021-03-13T15:17:20.900 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 2.1050000000000002e-06 2021-03-14T06:23:24.177 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.940 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "queued_for_pg", , 2021-03-13T15:17:20.940 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:13:52.857210+0000", , 2021-03-13T15:17:20.940 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 8.3113000000000002e-05 2021-03-14T06:23:24.178 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.963 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "op_commit", , 2021-03-13T15:17:20.963 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:14:52.370399+0000", , 2021-03-13T15:17:20.963 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 8.6979999999999997e-05 2021-03-14T06:23:24.179 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:20.986 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:20.986 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "throttled", , 2021-03-13T15:17:20.987 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:15:27.022610+0000", 2021-03-14T06:23:24.181 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.035 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.035 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "queued_for_pg", , 2021-03-13T15:17:21.035 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:15:27.328818+0000", 2021-03-14T06:23:24.182 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.057 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.057 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.058 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "header_read", 2021-03-14T06:23:24.183 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.079 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.079 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.080 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "all_read", 2021-03-14T06:23:24.184 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.102 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.103 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.103 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "all_read", 2021-03-14T06:23:24.185 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.141 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.142 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.142 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "sub_op_commit_rec", 2021-03-14T06:23:24.186 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.164 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.164 INFO:teuthology.orchestra.run.gibba007.stdout: "events": [ , 2021-03-13T15:17:21.164 INFO:teuthology.orchestra.run.gibba007.stdout: { 2021-03-14T06:23:24.187 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.187 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 0 , 2021-03-13T15:17:21.187 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.187 INFO:teuthology.orchestra.run.gibba007.stdout: { 2021-03-14T06:23:24.188 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.209 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 2.7654999999999999e-05 , 2021-03-13T15:17:21.210 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.210 INFO:teuthology.orchestra.run.gibba007.stdout: { 2021-03-14T06:23:24.190 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.232 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 2.5986000000000002e-05 , 2021-03-13T15:17:21.232 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.232 INFO:teuthology.orchestra.run.gibba007.stdout: { 2021-03-14T06:23:24.191 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.538 INFO:teuthology.orchestra.run.gibba007.stdout:} , 2021-03-13T15:17:21.545 DEBUG:teuthology.orchestra.run.gibba007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops , 2021-03-13T15:17:21.555+0000 7fd1837f1700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 35107) UID: 0 2021-03-14T06:23:24.192 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.713 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.713 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "started", , 2021-03-13T15:17:21.714 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:12:13.737926+0000", 2021-03-14T06:23:24.193 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.760 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.764 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "sub_op_commit_rec", , 2021-03-13T15:17:21.765 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:12:14.051088+0000", 2021-03-14T06:23:24.194 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.814 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.814 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "done", , 2021-03-13T15:17:21.815 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:14:17.728322+0000", 2021-03-14T06:23:24.195 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.852 INFO:teuthology.orchestra.run.gibba007.stdout: "events": [ , 2021-03-13T15:17:21.852 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.852 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "initiated", 2021-03-14T06:23:24.196 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.879 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.879 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.879 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "all_read", 2021-03-14T06:23:24.197 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.904 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.904 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.904 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "queued_for_pg", 2021-03-14T06:23:24.199 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.932 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.933 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.933 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "waiting for readable", 2021-03-14T06:23:24.200 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.958 INFO:teuthology.orchestra.run.gibba007.stdout: } , 2021-03-13T15:17:21.958 INFO:teuthology.orchestra.run.gibba007.stdout: ] , 2021-03-13T15:17:21.958 INFO:teuthology.orchestra.run.gibba007.stdout: } 2021-03-14T06:23:24.201 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:21.982 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:21.982 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:21.982 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "done", 2021-03-14T06:23:24.202 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.004 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:22.004 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:22.004 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "sub_op_commit_rec", 2021-03-14T06:23:24.203 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.026 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:22.027 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:22.027 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "started", 2021-03-14T06:23:24.203 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.050 INFO:teuthology.orchestra.run.gibba007.stdout: }, , 2021-03-13T15:17:22.051 INFO:teuthology.orchestra.run.gibba007.stdout: { , 2021-03-13T15:17:22.051 INFO:teuthology.orchestra.run.gibba007.stdout: "event": "reached_pg", 2021-03-14T06:23:24.204 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.088 INFO:teuthology.orchestra.run.gibba007.stdout: "flag_point": "started", , 2021-03-13T15:17:22.088 INFO:teuthology.orchestra.run.gibba007.stdout: "events": [ , 2021-03-13T15:17:22.088 INFO:teuthology.orchestra.run.gibba007.stdout: { 2021-03-14T06:23:24.205 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.111 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 0.61487767699999996 , 2021-03-13T15:17:22.111 INFO:teuthology.orchestra.run.gibba007.stdout: } , 2021-03-13T15:17:22.111 INFO:teuthology.orchestra.run.gibba007.stdout: ] 2021-03-14T06:23:24.205 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:22.138 INFO:teuthology.orchestra.run.gibba007.stdout: "time": "2021-03-13T15:16:36.845235+0000", , 2021-03-13T15:17:22.139 INFO:teuthology.orchestra.run.gibba007.stdout: "duration": 7.5050000000000001e-06 , 2021-03-13T15:17:22.139 INFO:teuthology.orchestra.run.gibba007.stdout: }, 2021-03-14T06:23:24.206 WARNING:scrape:Ignoring malparsed backtrace: suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) 2021-03-14T06:23:24.207 WARNING:scrape:Ignoring malparsed backtrace: mon.b: mon_osd_down_out_interval = '600' (not observed, change may require restart) , mon.c: mon_osd_down_out_interval = '600' (not observed, change may require restart) , 2021-03-13T15:17:25.014 DEBUG:teuthology.orchestra.run.gibba007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2021-03-14T06:23:24.209 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:46.942 DEBUG:teuthology.orchestra.run.gibba007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-13T15:17:47.164 INFO:teuthology.orchestra.run.gibba007.stdout: , dumped all 2021-03-14T06:23:24.210 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:17:52.248 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2021-03-13T15:16:25.517251+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=16, tm_sec=25, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=17, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T15:17:52.249 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2021-03-13T15:17:40.166396+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=17, tm_sec=40, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=17, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T15:17:52.249 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2021-03-13T15:17:47.067338+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=17, tm_sec=47, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=15, tm_min=17, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:24.211 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:18:33.526 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-13T15:18:33.575 INFO:tasks.ceph.mon.c:Stopped , 2021-03-13T15:18:33.576 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-03-14T06:23:24.212 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T15:19:18.112 INFO:teuthology.orchestra.run.gibba007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample , 2021-03-13T15:19:18.113 INFO:teuthology.orchestra.run.gibba007.stdout:=============================================================================== , 2021-03-13T15:19:18.113 INFO:teuthology.orchestra.run.gibba007.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 2 -204us[ -204us] +/- 104ms 2021-03-14T06:23:24.213 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-03-14T06:23:25.645 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f4a6085d308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f4a6085d522] 2021-03-14T06:23:25.646 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.325 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "initiated", , 2021-03-13T21:08:52.325 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:06:25.062074+0000", , 2021-03-13T21:08:52.326 INFO:teuthology.orchestra.run.gibba004.stdout: "duration": 0 2021-03-14T06:23:25.646 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.381 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "commit_sent", , 2021-03-13T21:08:52.381 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:06:25.223563+0000", , 2021-03-13T21:08:52.381 INFO:teuthology.orchestra.run.gibba004.stdout: "duration": 2.6122000000000002e-05 2021-03-14T06:23:25.647 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.448 INFO:teuthology.orchestra.run.gibba004.stdout: { , 2021-03-13T21:08:52.448 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "dispatched", , 2021-03-13T21:08:52.448 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:06:26.339574+0000", 2021-03-14T06:23:25.648 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.494 INFO:teuthology.orchestra.run.gibba004.stdout: "events": [ , 2021-03-13T21:08:52.494 INFO:teuthology.orchestra.run.gibba004.stdout: { , 2021-03-13T21:08:52.494 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "initiated", 2021-03-14T06:23:25.648 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.558 INFO:teuthology.orchestra.run.gibba004.stdout: }, , 2021-03-13T21:08:52.558 INFO:teuthology.orchestra.run.gibba004.stdout: { , 2021-03-13T21:08:52.558 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "commit_sent", 2021-03-14T06:23:25.649 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.605 INFO:teuthology.orchestra.run.gibba004.stdout: "duration": 0.068081374 , 2021-03-13T21:08:52.605 INFO:teuthology.orchestra.run.gibba004.stdout: }, , 2021-03-13T21:08:52.605 INFO:teuthology.orchestra.run.gibba004.stdout: { 2021-03-14T06:23:25.650 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.647 INFO:teuthology.orchestra.run.gibba004.stdout: "type_data": { , 2021-03-13T21:08:52.647 INFO:teuthology.orchestra.run.gibba004.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T21:08:52.647 INFO:teuthology.orchestra.run.gibba004.stdout: "events": [ 2021-03-14T06:23:25.650 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.712 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:07:21.742848+0000", , 2021-03-13T21:08:52.712 INFO:teuthology.orchestra.run.gibba004.stdout: "duration": 0.090103774999999997 , 2021-03-13T21:08:52.713 INFO:teuthology.orchestra.run.gibba004.stdout: } 2021-03-14T06:23:25.651 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.742 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "done", , 2021-03-13T21:08:52.742 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:07:31.755435+0000", , 2021-03-13T21:08:52.742 INFO:teuthology.orchestra.run.gibba004.stdout: "duration": 0.089409889000000006 2021-03-14T06:23:25.651 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.800 INFO:teuthology.orchestra.run.gibba004.stdout: { , 2021-03-13T21:08:52.801 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "done", , 2021-03-13T21:08:52.801 INFO:teuthology.orchestra.run.gibba004.stdout: "time": "2021-03-13T21:07:36.586913+0000", 2021-03-14T06:23:25.652 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:52.837 INFO:teuthology.orchestra.run.gibba004.stdout: }, , 2021-03-13T21:08:52.837 INFO:teuthology.orchestra.run.gibba004.stdout: { , 2021-03-13T21:08:52.837 INFO:teuthology.orchestra.run.gibba004.stdout: "event": "reached_pg", 2021-03-14T06:23:25.653 WARNING:scrape:Ignoring malparsed backtrace: osd.1: osd_enable_op_tracker = 'false' , osd.2: osd_enable_op_tracker = 'false' , osd.3: osd_enable_op_tracker = 'false' 2021-03-14T06:23:25.654 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:08:56.773 INFO:teuthology.orchestra.run.gibba004.stdout:mon.a: {} , 2021-03-13T21:08:56.787 INFO:teuthology.orchestra.run.gibba004.stdout:mon.b: {} , 2021-03-13T21:08:56.799 INFO:teuthology.orchestra.run.gibba004.stdout:mon.c: {} 2021-03-14T06:23:25.658 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:09:17.605 INFO:teuthology.orchestra.run.gibba004.stdout: , dumped all , 2021-03-13T21:09:17.615 INFO:teuthology.orchestra.run.gibba004.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2021-03-13T21:09:16.603518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1793774368,"num_objects":662,"num_object_clones":31,"num_object_copies":1986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":171,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":28,"num_read":2604,"num_read_kb":1468546,"num_write":3311,"num_write_kb":1387036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":2151559500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":171,"num_flush_kb":468075,"num_evict":102,"num_evict_kb":255633,"num_promote":322,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4826,"ondisk_log_size":4826,"up":51,"acting":49,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":55,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4248392,"kb_used_data":4069576,"kb_used_omap":0,"kb_used_meta":178752,"kb_avail":750726328,"statfs":{"total":773094113280,"available":768743759872,"internally_reserved":0,"allocated":4167245824,"data_stored":4275098006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":183042048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":-47,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":139,"num_bytes_recovered":197273325,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.735368"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":"63","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603042+0000","last_change":"2021-03-13T21:08:56.607993+0000","last_active":"2021-03-13T21:09:11.603042+0000","last_peered":"2021-03-13T21:09:11.603042+0000","last_clean":"2021-03-13T21:09:11.603042+0000","last_became_active":"2021-03-13T21:08:56.607704+0000","last_became_peered":"2021-03-13T21:08:56.607704+0000","last_unstale":"2021-03-13T21:09:11.603042+0000","last_undegraded":"2021-03-13T21:09:11.603042+0000","last_fullsized":"2021-03-13T21:09:11.603042+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:45.395709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:45.395709+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"188'437","reported_seq":"1717","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603039+0000","last_change":"2021-03-13T21:09:10.614893+0000","last_active":"2021-03-13T21:09:11.603039+0000","last_peered":"2021-03-13T21:09:11.603039+0000","last_clean":"2021-03-13T21:09:11.603039+0000","last_became_active":"2021-03-13T21:09:08.611449+0000","last_became_peered":"2021-03-13T21:09:08.611449+0000","last_unstale":"2021-03-13T21:09:11.603039+0000","last_undegraded":"2021-03-13T21:09:11.603039+0000","last_fullsized":"2021-03-13T21:09:11.603039+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'315","last_scrub_stamp":"2021-03-13T21:07:57.678564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:15.913215+0000","last_clean_scrub_stamp":"2021-03-13T21:07:57.678564+0000","log_size":437,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":116515805,"num_objects":47,"num_object_clones":4,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":5,"num_read":153,"num_read_kb":111612,"num_write":213,"num_write_kb":38392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":318,"num_bytes_recovered":424753749,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":41,"num_flush_kb":112269,"num_evict":24,"num_evict_kb":61643,"num_promote":77,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":"26","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:12.722549+0000","last_change":"2021-03-13T21:08:57.610107+0000","last_active":"2021-03-13T21:09:12.722549+0000","last_peered":"2021-03-13T21:09:12.722549+0000","last_clean":"2021-03-13T21:09:12.722549+0000","last_became_active":"2021-03-13T21:08:57.609910+0000","last_became_peered":"2021-03-13T21:08:57.609910+0000","last_unstale":"2021-03-13T21:09:12.722549+0000","last_undegraded":"2021-03-13T21:09:12.722549+0000","last_fullsized":"2021-03-13T21:09:12.722549+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:44.884332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:44.884332+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3","version":"173'632","reported_seq":"2196","reported_epoch":"192","state":"active+undersized+remapped+backfill_wait","last_fresh":"2021-03-13T21:09:12.872754+0000","last_change":"2021-03-13T21:09:12.872754+0000","last_active":"2021-03-13T21:09:12.872754+0000","last_peered":"2021-03-13T21:09:12.872754+0000","last_clean":"2021-03-13T21:08:54.261617+0000","last_became_active":"2021-03-13T21:08:58.861793+0000","last_became_peered":"2021-03-13T21:08:58.861793+0000","last_unstale":"2021-03-13T21:09:12.872754+0000","last_undegraded":"2021-03-13T21:09:12.872754+0000","last_fullsized":"2021-03-13T21:08:58.609960+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'512","last_scrub_stamp":"2021-03-13T21:07:48.748299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:13.904700+0000","last_clean_scrub_stamp":"2021-03-13T21:07:48.748299+0000","log_size":632,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"stat_sum":{"num_bytes":286861470,"num_objects":103,"num_object_clones":6,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":103,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":1,"num_read":415,"num_read_kb":187555,"num_write":517,"num_write_kb":271972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":557394926,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[1,6],"avail_no_missing":["1","0","3","5","6"],"object_location_counts":[{"shards":"1,6","objects":103}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"221","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.940116+0000","last_change":"2021-03-13T21:09:11.939084+0000","last_active":"2021-03-13T21:09:11.940116+0000","last_peered":"2021-03-13T21:09:11.940116+0000","last_clean":"2021-03-13T21:09:11.940116+0000","last_became_active":"2021-03-13T21:08:56.606330+0000","last_became_peered":"2021-03-13T21:08:56.606330+0000","last_unstale":"2021-03-13T21:09:11.940116+0000","last_undegraded":"2021-03-13T21:09:11.940116+0000","last_fullsized":"2021-03-13T21:09:11.940116+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:13.024304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:13.024304+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"181'733","reported_seq":"2061","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603054+0000","last_change":"2021-03-13T21:09:09.612068+0000","last_active":"2021-03-13T21:09:11.603054+0000","last_peered":"2021-03-13T21:09:11.603054+0000","last_clean":"2021-03-13T21:09:11.603054+0000","last_became_active":"2021-03-13T21:08:54.661459+0000","last_became_peered":"2021-03-13T21:08:54.661459+0000","last_unstale":"2021-03-13T21:09:11.603054+0000","last_undegraded":"2021-03-13T21:09:11.603054+0000","last_fullsized":"2021-03-13T21:09:11.603054+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'566","last_scrub_stamp":"2021-03-13T21:07:49.025181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:13.904700+0000","last_clean_scrub_stamp":"2021-03-13T21:07:49.025181+0000","log_size":733,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":328787348,"num_objects":114,"num_object_clones":3,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":114,"num_whiteouts":1,"num_read":489,"num_read_kb":219584,"num_write":602,"num_write_kb":309105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":319,"num_bytes_recovered":414307167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5f","length":"1"},{"start":"66","length":"2"},{"start":"72","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":"201","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:12.965445+0000","last_change":"2021-03-13T21:09:09.830180+0000","last_active":"2021-03-13T21:09:12.965445+0000","last_peered":"2021-03-13T21:09:12.965445+0000","last_clean":"2021-03-13T21:09:12.965445+0000","last_became_active":"2021-03-13T21:06:04.884914+0000","last_became_peered":"2021-03-13T21:06:04.884914+0000","last_unstale":"2021-03-13T21:09:12.965445+0000","last_undegraded":"2021-03-13T21:09:12.965445+0000","last_fullsized":"2021-03-13T21:09:12.965445+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:09:09.830133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:09:09.830133+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"139","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:12.965426+0000","last_change":"2021-03-13T21:07:45.767410+0000","last_active":"2021-03-13T21:09:12.965426+0000","last_peered":"2021-03-13T21:09:12.965426+0000","last_clean":"2021-03-13T21:09:12.965426+0000","last_became_active":"2021-03-13T21:07:06.628950+0000","last_became_peered":"2021-03-13T21:07:06.628950+0000","last_unstale":"2021-03-13T21:09:12.965426+0000","last_undegraded":"2021-03-13T21:09:12.965426+0000","last_fullsized":"2021-03-13T21:09:12.965426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:45.767355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:02.865770+0000","last_clean_scrub_stamp":"2021-03-13T21:07:45.767355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"172'767","reported_seq":"1449","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603443+0000","last_change":"2021-03-13T21:08:48.618870+0000","last_active":"2021-03-13T21:09:11.603443+0000","last_peered":"2021-03-13T21:09:11.603443+0000","last_clean":"2021-03-13T21:09:11.603443+0000","last_became_active":"2021-03-13T21:06:14.919741+0000","last_became_peered":"2021-03-13T21:06:14.919741+0000","last_unstale":"2021-03-13T21:09:11.603443+0000","last_undegraded":"2021-03-13T21:09:11.603443+0000","last_fullsized":"2021-03-13T21:09:11.603443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'626","last_scrub_stamp":"2021-03-13T21:07:46.093870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:13.904700+0000","last_clean_scrub_stamp":"2021-03-13T21:07:46.093870+0000","log_size":767,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":351730897,"num_objects":125,"num_object_clones":2,"num_object_copies":375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125,"num_whiteouts":0,"num_read":473,"num_read_kb":207927,"num_write":642,"num_write_kb":302287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"5f","length":"1"},{"start":"66","length":"2"},{"start":"72","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":"270","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:13.605857+0000","last_change":"2021-03-13T21:09:13.606512+0000","last_active":"2021-03-13T21:09:13.605857+0000","last_peered":"2021-03-13T21:09:13.605857+0000","last_clean":"2021-03-13T21:09:13.605857+0000","last_became_active":"2021-03-13T21:08:56.605627+0000","last_became_peered":"2021-03-13T21:08:56.605627+0000","last_unstale":"2021-03-13T21:09:13.605857+0000","last_undegraded":"2021-03-13T21:09:13.605857+0000","last_fullsized":"2021-03-13T21:09:13.605857+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:20.481648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:20.481648+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"167'751","reported_seq":"1964","reported_epoch":"192","state":"active+recovering+undersized+remapped","last_fresh":"2021-03-13T21:09:13.236582+0000","last_change":"2021-03-13T21:09:07.624480+0000","last_active":"2021-03-13T21:09:13.236582+0000","last_peered":"2021-03-13T21:09:13.236582+0000","last_clean":"2021-03-13T21:09:05.068815+0000","last_became_active":"2021-03-13T21:09:07.618144+0000","last_became_peered":"2021-03-13T21:09:07.618144+0000","last_unstale":"2021-03-13T21:09:13.236582+0000","last_undegraded":"2021-03-13T21:09:13.236582+0000","last_fullsized":"2021-03-13T21:09:07.608271+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'671","last_scrub_stamp":"2021-03-13T21:08:22.423761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:13.904700+0000","last_clean_scrub_stamp":"2021-03-13T21:08:22.423761+0000","log_size":751,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":367550563,"num_objects":129,"num_object_clones":4,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":68,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":2,"num_read":567,"num_read_kb":346965,"num_write":627,"num_write_kb":307973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":359229668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":61},{"shards":"0,3,5","objects":68}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5f","length":"1"},{"start":"66","length":"2"},{"start":"72","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"4.3","version":"178'493","reported_seq":"1066","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.637819+0000","last_change":"2021-03-13T21:09:01.329724+0000","last_active":"2021-03-13T21:09:11.637819+0000","last_peered":"2021-03-13T21:09:11.637819+0000","last_clean":"2021-03-13T21:09:11.637819+0000","last_became_active":"2021-03-13T21:08:56.605566+0000","last_became_peered":"2021-03-13T21:08:56.605566+0000","last_unstale":"2021-03-13T21:09:11.637819+0000","last_undegraded":"2021-03-13T21:09:11.637819+0000","last_fullsized":"2021-03-13T21:09:11.637819+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'438","last_scrub_stamp":"2021-03-13T21:08:25.267268+0000","last_deep_scrub":"145'438","last_deep_scrub_stamp":"2021-03-13T21:08:25.267268+0000","last_clean_scrub_stamp":"2021-03-13T21:08:25.267268+0000","log_size":493,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":105876321,"num_objects":47,"num_object_clones":4,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":7,"num_read":184,"num_read_kb":151612,"num_write":236,"num_write_kb":47859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":118788893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":42,"num_flush_kb":114748,"num_evict":19,"num_evict_kb":44293,"num_promote":73,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":"236","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:12.011773+0000","last_change":"2021-03-13T21:08:57.610774+0000","last_active":"2021-03-13T21:09:12.011773+0000","last_peered":"2021-03-13T21:09:12.011773+0000","last_clean":"2021-03-13T21:09:12.011773+0000","last_became_active":"2021-03-13T21:08:57.610490+0000","last_became_peered":"2021-03-13T21:08:57.610490+0000","last_unstale":"2021-03-13T21:09:12.011773+0000","last_undegraded":"2021-03-13T21:09:12.011773+0000","last_fullsized":"2021-03-13T21:09:12.011773+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:08:23.461171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:08:23.461171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"54","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603051+0000","last_change":"2021-03-13T21:08:57.611544+0000","last_active":"2021-03-13T21:09:11.603051+0000","last_peered":"2021-03-13T21:09:11.603051+0000","last_clean":"2021-03-13T21:09:11.603051+0000","last_became_active":"2021-03-13T21:08:57.611277+0000","last_became_peered":"2021-03-13T21:08:57.611277+0000","last_unstale":"2021-03-13T21:09:11.603051+0000","last_undegraded":"2021-03-13T21:09:11.603051+0000","last_fullsized":"2021-03-13T21:09:11.603051+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:45.067601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:45.067601+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"172'544","reported_seq":"1165","reported_epoch":"191","state":"active+clean","last_fresh":"2021-03-13T21:09:11.603405+0000","last_change":"2021-03-13T21:08:58.182460+0000","last_active":"2021-03-13T21:09:11.603405+0000","last_peered":"2021-03-13T21:09:11.603405+0000","last_clean":"2021-03-13T21:09:11.603405+0000","last_became_active":"2021-03-13T21:08:57.612098+0000","last_became_peered":"2021-03-13T21:08:57.612098+0000","last_unstale":"2021-03-13T21:09:11.603405+0000","last_undegraded":"2021-03-13T21:09:11.603405+0000","last_fullsized":"2021-03-13T21:09:11.603405+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'348","last_scrub_stamp":"2021-03-13T21:07:56.004737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:15.913215+0000","last_clean_scrub_stamp":"2021-03-13T21:07:56.004737+0000","log_size":544,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110794252,"num_objects":50,"num_object_clones":4,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":9,"num_read":180,"num_read_kb":140697,"num_write":243,"num_write_kb":57720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":16246104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":49,"num_flush_kb":130166,"num_evict":28,"num_evict_kb":75184,"num_promote":88,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7a","length":"1"},{"start":"7e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":"190","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:12.955521+0000","last_change":"2021-03-13T21:07:48.052395+0000","last_active":"2021-03-13T21:09:12.955521+0000","last_peered":"2021-03-13T21:09:12.955521+0000","last_clean":"2021-03-13T21:09:12.955521+0000","last_became_active":"2021-03-13T21:06:04.885114+0000","last_became_peered":"2021-03-13T21:06:04.885114+0000","last_unstale":"2021-03-13T21:09:12.955521+0000","last_undegraded":"2021-03-13T21:09:12.955521+0000","last_fullsized":"2021-03-13T21:09:12.955521+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T21:07:48.052342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:03.870722+0000","last_clean_scrub_stamp":"2021-03-13T21:07:48.052342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"172'469","reported_seq":"1318","reported_epoch":"192","state":"active+clean","last_fresh":"2021-03-13T21:09:12.722578+0000","last_change":"2021-03-13T21:09:10.063766+0000","last_active":"2021-03-13T21:09:12.722578+0000","last_peered":"2021-03-13T21:09:12.722578+0000","last_clean":"2021-03-13T21:09:12.722578+0000","last_became_active":"2021-03-13T21:09:10.061868+0000","last_became_peered":"2021-03-13T21:09:10.061868+0000","last_unstale":"2021-03-13T21:09:12.722578+0000","last_undegraded":"2021-03-13T21:09:12.722578+0000","last_fullsized":"2021-03-13T21:09:12.722578+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'311","last_scrub_stamp":"2021-03-13T21:07:59.784894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T21:06:15.913215+0000","last_clean_scrub_stamp":"2021-03-13T21:07:59.784894+0000","log_size":469,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":125657712,"num_objects":47,"num_object_clones":4,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":143,"num_read_kb":102594,"num_write":231,"num_write_kb":51728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":260838993,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":39,"num_flush_kb":110892,"num_evict":31,"num_evict_kb":74513,"num_promote":84,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7a","length":"1"},{"start":"7e","length":"1"}]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":458844090,"num_objects":191,"num_object_clones":16,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":24,"num_read":660,"num_read_kb":506515,"num_write":923,"num_write_kb":195699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":657,"num_bytes_recovered":820627739,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":171,"num_flush_kb":468075,"num_evict":102,"num_evict_kb":255633,"num_promote":322,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1303506944,"data_stored":1388027167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1943,"ondisk_log_size":1943,"up":12,"acting":12,"num_store_stats":8},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1334930278,"num_objects":471,"num_object_clones":15,"num_object_copies":1413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":171,"num_objects_unfound":0,"num_objects_dirty":471,"num_whiteouts":4,"num_read":1944,"num_read_kb":962031,"num_write":2388,"num_write_kb":1191337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1029,"num_bytes_recovered":1330931761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2934931456,"data_stored":2967734605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2883,"ondisk_log_size":2883,"up":12,"acting":10,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":208476,"kb_used_data":193236,"kb_used_omap":0,"kb_used_meta":15232,"kb_avail":94163364,"statfs":{"total":96636764160,"available":96423284736,"internally_reserved":0,"allocated":197873664,"data_stored":217432284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":15597568},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":49.347000000000001,"5min":31.260999999999999,"15min":29.309000000000001},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":973.39499999999998,"5min":973.39499999999998,"15min":973.39499999999998},"last":0.54300000000000004},{"interface":"front","average":{"1min":37.518000000000001,"5min":13.972,"15min":11.662000000000001},"min":{"1min":0.46600000000000003,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":734.44299999999998,"5min":734.44299999999998,"15min":734.44299999999998},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.86299999999999999,"15min":0.86099999999999999},"min":{"1min":0.48399999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.3100000000000001,"5min":1.321,"15min":1.321},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.754,"5min":6.649,"15min":9.2509999999999994},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3759999999999999,"5min":175.08799999999999,"15min":175.08799999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":4.7990000000000004,"15min":6.5780000000000003},"min":{"1min":0.50700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.258,"5min":122.511,"15min":122.511},"last":413.46699999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":3.3969999999999998,"15min":4.5549999999999997},"min":{"1min":0.48899999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.244,"5min":78.709000000000003,"15min":78.709000000000003},"last":0.70299999999999996}]},{"osd":3,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.89900000000000002,"15min":0.88100000000000001},"min":{"1min":0.63300000000000001,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.393,"5min":1.393,"15min":1.393},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":0.95199999999999996,"15min":0.95999999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2809999999999999,"5min":1.363,"15min":1.363},"last":0.97699999999999998}]},{"osd":4,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.78600000000000003,"15min":0.78400000000000003},"min":{"1min":0.53300000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.212,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":5.4610000000000003,"15min":6.835},"min":{"1min":0.627,"5min":0.37,"15min":0.37},"max":{"1min":1.1240000000000001,"5min":122.47799999999999,"15min":122.47799999999999},"last":1.0229999999999999}]},{"osd":5,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":23.082000000000001,"15min":32.941000000000003},"min":{"1min":0.67600000000000005,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.29,"5min":665.94600000000003,"15min":665.94600000000003},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.88,"5min":24.449999999999999,"15min":34.917000000000002},"min":{"1min":0.69099999999999995,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.341,"5min":706.38,"15min":706.38},"last":1.111}]},{"osd":6,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.88500000000000001,"15min":0.88300000000000001},"min":{"1min":0.68000000000000005,"5min":0.62,"15min":0.62},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":1.139},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.84799999999999998,"15min":0.82599999999999996},"min":{"1min":0.72999999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"last":1.1899999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":751388,"kb_used_data":728852,"kb_used_omap":0,"kb_used_meta":22528,"kb_avail":93620452,"statfs":{"total":96636764160,"available":95867342848,"internally_reserved":0,"allocated":746344448,"data_stored":761232519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23068672},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.98299999999999998,"15min":0.88400000000000001},"min":{"1min":0.42399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.2689999999999999,"5min":16.119,"15min":16.119},"last":0.504},{"interface":"front","average":{"1min":0.69699999999999995,"5min":20.663,"15min":29.515999999999998},"min":{"1min":0.41299999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1739999999999999,"5min":591.96600000000001,"15min":591.96600000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.82199999999999995,"15min":0.80700000000000005},"min":{"1min":0.57099999999999995,"5min":0.378,"15min":0.378},"max":{"1min":1.1970000000000001,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.82699999999999996,"15min":0.79400000000000004},"min":{"1min":0.55800000000000005,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.3,"5min":1.425,"15min":1.425},"last":0.91900000000000004}]},{"osd":2,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":4.2949999999999999,"15min":1.9299999999999999},"min":{"1min":0.51500000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0289999999999999,"5min":373.69799999999998,"15min":373.69799999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2090000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"last":1.024}]},{"osd":3,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.90900000000000003,"15min":0.93100000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.2230000000000001,"5min":1.407,"15min":1.407},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.88400000000000001,"15min":0.88100000000000001},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.276,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.72299999999999998}]},{"osd":4,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":29.992999999999999,"15min":42.929000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.179,"5min":971.78599999999994,"15min":971.78599999999994},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77100000000000002,"15min":0.76300000000000001},"min":{"1min":0.53000000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.93300000000000005}]},{"osd":5,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76900000000000002,"15min":0.752},"min":{"1min":0.50900000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.2010000000000001,"5min":1.246,"15min":1.246},"last":0.878},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90300000000000002,"15min":0.89300000000000002},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":1.2310000000000001,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0800000000000001}]},{"osd":7,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.93200000000000005,"15min":0.93799999999999994},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.246,"5min":1.359,"15min":1.359},"last":1.0049999999999999},{"interface":"front","average":{"1min":17.338999999999999,"5min":30.969999999999999,"15min":40.659999999999997},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":347.00400000000002,"5min":893.12199999999996,"15min":893.12199999999996},"last":0.90400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":529396,"kb_used_data":506604,"kb_used_omap":0,"kb_used_meta":22784,"kb_avail":93842444,"statfs":{"total":96636764160,"available":96094662656,"internally_reserved":0,"allocated":518762496,"data_stored":528293281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23330816},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.60799999999999998,"15min":0.62},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96199999999999997,"5min":1.117,"15min":1.117},"last":0.55800000000000005},{"interface":"front","average":{"1min":31.928999999999998,"5min":6.8879999999999999,"15min":2.7040000000000002},"min":{"1min":0.435,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":563.81600000000003,"5min":563.81600000000003,"15min":563.81600000000003},"last":0.60799999999999998}]},{"osd":2,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.501,"15min":0.46999999999999997},"min":{"1min":0.34699999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.86399999999999999,"5min":1.032,"15min":1.032},"last":0.50700000000000001},{"interface":"front","average":{"1min":33.631999999999998,"5min":7.21,"15min":2.7930000000000001},"min":{"1min":0.52000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":500.86500000000001,"5min":500.86500000000001,"15min":500.86500000000001},"last":0.67800000000000005}]},{"osd":3,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.624,"15min":0.628},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.05,"5min":1.1419999999999999,"15min":1.1419999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":45.302,"5min":9.484,"15min":3.5099999999999998},"min":{"1min":0.40300000000000002,"5min":0.379,"15min":0.379},"max":{"1min":805.495,"5min":805.495,"15min":805.495},"last":0.58599999999999997}]},{"osd":4,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.181,"15min":1.256},"min":{"1min":0.63900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.395,"5min":3.6269999999999998,"15min":3.6269999999999998},"last":1.3460000000000001},{"interface":"front","average":{"1min":0.95199999999999996,"5min":19.047999999999998,"15min":18.655999999999999},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.371,"5min":682.21100000000001,"15min":682.21100000000001},"last":1.335}]},{"osd":5,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.05,"15min":1.087},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.3169999999999999,"5min":2.956,"15min":2.956},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.008,"15min":1.042},"min":{"1min":0.57399999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.377,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":1.175}]},{"osd":6,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":21.238,"15min":30.212},"min":{"1min":0.68700000000000006,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.3600000000000001,"5min":705.84100000000001,"15min":705.84100000000001},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.002,"5min":14.173,"15min":20.009},"min":{"1min":0.69899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.409,"5min":459.03800000000001,"15min":459.03800000000001},"last":1.4159999999999999}]},{"osd":7,"last update":"Sat Mar 13 21:09:06 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":3.8610000000000002,"15min":1.9450000000000001},"min":{"1min":0.67400000000000004,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2889999999999999,"5min":260.48700000000002,"15min":260.48700000000002},"last":1.214},{"interface":"front","average":{"1min":0.94199999999999995,"5min":10.305999999999999,"15min":4.1769999999999996},"min":{"1min":0.61699999999999999,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.28,"5min":832.99800000000005,"15min":832.99800000000005},"last":1.131}]}]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":461632,"kb_used_data":430904,"kb_used_omap":0,"kb_used_meta":30720,"kb_avail":93910208,"statfs":{"total":96636764160,"available":96164052992,"internally_reserved":0,"allocated":441245696,"data_stored":459769168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31457280},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.623},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.879,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59899999999999998,"15min":0.59499999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.47399999999999998}]},{"osd":2,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":33.540999999999997,"5min":39.945999999999998,"15min":50.122999999999998},"min":{"1min":0.34399999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":660.17999999999995,"5min":660.17999999999995,"15min":660.17999999999995},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":5.9100000000000001,"15min":2.3130000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83499999999999996,"5min":619.66499999999996,"15min":619.66499999999996},"last":0.36099999999999999}]},{"osd":3,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.34799999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.63700000000000001,"5min":6.2699999999999996,"15min":8.7759999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.93000000000000005,"5min":174.678,"15min":174.678},"last":0.503}]},{"osd":4,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.048,"15min":1.0449999999999999},"min":{"1min":0.70399999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.4930000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":1.351},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.0109999999999999,"15min":1.0089999999999999},"min":{"1min":0.64700000000000002,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.157}]},{"osd":5,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":24.879999999999999,"15min":30.777999999999999},"min":{"1min":0.74299999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.5369999999999999,"5min":625.31700000000001,"15min":625.31700000000001},"last":1.337},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.018,"15min":1.008},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.4770000000000001,"5min":3.2839999999999998,"15min":3.2839999999999998},"last":1.298}]},{"osd":6,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.94299999999999995,"15min":0.93899999999999995},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.448,"5min":1.448,"15min":1.448},"last":1.2290000000000001},{"interface":"front","average":{"1min":40.642000000000003,"5min":8.875,"15min":3.5600000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":795.67100000000005,"5min":795.67100000000005,"15min":795.67100000000005},"last":0.99299999999999999}]},{"osd":7,"last update":"Sat Mar 13 21:09:10 2021","interfaces":[{"interface":"back","average":{"1min":8.2669999999999995,"5min":10.294,"15min":4.1079999999999997},"min":{"1min":0.68799999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":146.816,"5min":901.447,"15min":901.447},"last":1.1299999999999999},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.97099999999999997,"15min":0.96699999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":1.246}]}]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":227916,"kb_used_data":214660,"kb_used_omap":0,"kb_used_meta":13248,"kb_avail":94143924,"statfs":{"total":96636764160,"available":96403378176,"internally_reserved":0,"allocated":219811840,"data_stored":218152181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13565952},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.505,"15min":0.49099999999999999},"min":{"1min":0.28899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.0109999999999999,"5min":1.151,"15min":1.151},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.621,"15min":0.65100000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.95399999999999996,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.58899999999999997}]},{"osd":1,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67600000000000005,"15min":0.69399999999999995},"min":{"1min":0.46600000000000003,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.109,"5min":1.131,"15min":1.131},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.69099999999999995,"15min":0.68700000000000006},"min":{"1min":0.438,"5min":0.434,"15min":0.434},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.629}]},{"osd":3,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.75900000000000001,"15min":0.79700000000000004},"min":{"1min":0.45400000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91200000000000003,"5min":3.46,"15min":3.46},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":20.263000000000002,"15min":18.015999999999998},"min":{"1min":0.42599999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.1679999999999999,"5min":985.56299999999999,"15min":985.56299999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.018,"15min":1.0049999999999999},"min":{"1min":0.79400000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.484,"5min":1.484,"15min":1.484},"last":1.423},{"interface":"front","average":{"1min":0.97899999999999998,"5min":8.8100000000000005,"15min":3.605},"min":{"1min":0.68100000000000005,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5589999999999999,"5min":782.56100000000004,"15min":782.56100000000004},"last":1.2549999999999999}]},{"osd":5,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":45.701999999999998,"5min":9.9510000000000005,"15min":3.9929999999999999},"min":{"1min":0.72399999999999998,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":804.87599999999998,"5min":804.87599999999998,"15min":804.87599999999998},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0329999999999999,"15min":1.0349999999999999},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.2190000000000001}]},{"osd":6,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":9.2059999999999995,"15min":3.7490000000000001},"min":{"1min":0.80600000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.417,"5min":777.62,"15min":777.62},"last":1.175},{"interface":"front","average":{"1min":55.802999999999997,"5min":11.978,"15min":4.6710000000000003},"min":{"1min":0.70199999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":986.93399999999997,"5min":986.93399999999997,"15min":986.93399999999997},"last":1.24}]},{"osd":7,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.107,"15min":1.1639999999999999},"min":{"1min":0.73099999999999998,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.3919999999999999,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":10.678000000000001,"15min":4.2699999999999996},"min":{"1min":0.68700000000000006,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.5329999999999999,"5min":963.39099999999996,"15min":963.39099999999996},"last":1.2070000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":760812,"kb_used_data":739300,"kb_used_omap":0,"kb_used_meta":21504,"kb_avail":93611028,"statfs":{"total":96636764160,"available":95857692672,"internally_reserved":0,"allocated":757043200,"data_stored":776850048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":22020096},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.307,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72099999999999997,"5min":1.95,"15min":1.95},"last":0.40999999999999998},{"interface":"front","average":{"1min":39.481999999999999,"5min":24.027999999999999,"15min":23.664999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":664.34799999999996,"5min":664.34799999999996,"15min":664.34799999999996},"last":0.49299999999999999}]},{"osd":1,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.436,"5min":0.378,"15min":0.378},"max":{"1min":0.70599999999999996,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.56299999999999994,"15min":0.57899999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":13.234,"15min":18.84},"min":{"1min":0.437,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93300000000000005,"5min":379.05900000000003,"15min":379.05900000000003},"last":0.73499999999999999}]},{"osd":4,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":44.540999999999997,"15min":63.923999999999999},"min":{"1min":0.64400000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.371,"5min":956.95799999999997,"15min":956.95799999999997},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.3089999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.0389999999999999}]},{"osd":5,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":1.6200000000000001,"15min":1.179},"min":{"1min":0.60099999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2070000000000001,"5min":55.128999999999998,"15min":55.128999999999998},"last":0.97799999999999998},{"interface":"front","average":{"1min":49.951000000000001,"5min":31.146999999999998,"15min":27.936},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":834.928,"5min":834.928,"15min":834.928},"last":0.94399999999999995}]},{"osd":6,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.92700000000000005,"15min":0.93999999999999995},"min":{"1min":0.60599999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.276,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94099999999999995,"15min":0.93799999999999994},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.2350000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.999}]},{"osd":7,"last update":"Sat Mar 13 21:09:09 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":31.213000000000001,"15min":36.924999999999997},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2250000000000001,"5min":700.18799999999999,"15min":700.18799999999999},"last":1.014},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.95799999999999996,"15min":0.95999999999999996},"min":{"1min":0.64100000000000001,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.256,"5min":1.351,"15min":1.351},"last":1.0680000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323868,"kb_used_data":300756,"kb_used_omap":0,"kb_used_meta":23104,"kb_avail":94047972,"statfs":{"total":96636764160,"available":96305123328,"internally_reserved":0,"allocated":307974144,"data_stored":306247412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23658496},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.748,"15min":0.77800000000000002},"min":{"1min":0.36899999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.97799999999999998,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.629},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.57399999999999995,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.339,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":0.68400000000000005}]},{"osd":1,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":13.936999999999999,"15min":19.797000000000001},"min":{"1min":0.48699999999999999,"5min":0.378,"15min":0.378},"max":{"1min":1.1759999999999999,"5min":482.56900000000002,"15min":482.56900000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.87,"15min":0.85999999999999999},"min":{"1min":0.629,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.3100000000000001,"5min":3.9729999999999999,"15min":3.9729999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":26.882000000000001,"5min":26.038,"15min":31.478000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":525.47400000000005,"5min":525.47400000000005,"15min":525.47400000000005},"last":0.505},{"interface":"front","average":{"1min":40.189999999999998,"5min":22.446000000000002,"15min":8.0470000000000006},"min":{"1min":0.48799999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":790.57399999999996,"5min":805.63800000000003,"15min":805.63800000000003},"last":0.73799999999999999}]},{"osd":3,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":8.9930000000000003,"15min":3.609},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.2969999999999999,"5min":437.62700000000001,"15min":437.62700000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.89900000000000002,"5min":0.872,"15min":0.88700000000000001},"min":{"1min":0.66600000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3240000000000001,"5min":1.3859999999999999,"15min":1.3859999999999999},"last":0.78900000000000003}]},{"osd":5,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.81100000000000005,"15min":0.78900000000000003},"min":{"1min":0.55600000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.208,"5min":1.208,"15min":1.208},"last":0.996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.83699999999999997,"15min":0.79100000000000004},"min":{"1min":0.65000000000000002,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.165,"5min":4.0179999999999998,"15min":4.0179999999999998},"last":1.0229999999999999}]},{"osd":6,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":67.986000000000004,"5min":14.222,"15min":5.2729999999999997},"min":{"1min":0.54400000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":663.81299999999999,"5min":663.81299999999999,"15min":663.81299999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.5700000000000001,"5min":0.98199999999999998,"15min":0.88500000000000001},"min":{"1min":0.53800000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":16.204999999999998,"5min":16.204999999999998,"15min":16.204999999999998},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Mar 13 21:09:07 2021","interfaces":[{"interface":"back","average":{"1min":14.861000000000001,"5min":3.649,"15min":1.798},"min":{"1min":0.59299999999999997,"5min":0.5,"15min":0.5},"max":{"1min":282.42899999999997,"5min":282.42899999999997,"15min":282.42899999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.94799999999999995,"15min":0.91500000000000004},"min":{"1min":0.67500000000000004,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.2769999999999999,"5min":4.0049999999999999,"15min":4.0049999999999999},"last":1.0489999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":984904,"kb_used_data":955264,"kb_used_omap":0,"kb_used_meta":29632,"kb_avail":93386936,"statfs":{"total":96636764160,"available":95628222464,"internally_reserved":0,"allocated":978190336,"data_stored":1007121113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30343168},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":30.693000000000001,"15min":43.984000000000002},"min":{"1min":0.50700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.298,"5min":949.03700000000003,"15min":949.03700000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.78300000000000003,"15min":0.79700000000000004},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.89400000000000002}]},{"osd":1,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":51.317999999999998,"5min":11.045999999999999,"15min":4.3440000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":807.78099999999995,"5min":807.78099999999995,"15min":807.78099999999995},"last":1.097},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.89600000000000002,"15min":0.89900000000000002},"min":{"1min":0.55100000000000005,"5min":0.502,"15min":0.502},"max":{"1min":1.381,"5min":2.7519999999999998,"15min":2.7519999999999998},"last":0.97599999999999998}]},{"osd":2,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":57.238,"5min":12.083,"15min":4.5549999999999997},"min":{"1min":0.47999999999999998,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":904.01499999999999,"5min":904.01499999999999,"15min":904.01499999999999},"last":1.028},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.54400000000000004,"5min":0.434,"15min":0.434},"max":{"1min":1.262,"5min":2.766,"15min":2.766},"last":0.92600000000000005}]},{"osd":3,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":2.4260000000000002,"15min":1.4199999999999999},"min":{"1min":0.64700000000000002,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.484,"5min":182.06200000000001,"15min":182.06200000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.98499999999999999,"15min":1.0009999999999999},"min":{"1min":0.70099999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.05}]},{"osd":4,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86899999999999999,"15min":0.86699999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.107,"5min":1.375,"15min":1.375},"last":1.3240000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.92600000000000005,"15min":0.93600000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.208,"5min":1.4179999999999999,"15min":1.4179999999999999},"last":1.133}]},{"osd":6,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":9.9789999999999992,"5min":2.6360000000000001,"15min":1.3999999999999999},"min":{"1min":0.53300000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":147.791,"5min":147.791,"15min":147.791},"last":1.3029999999999999},{"interface":"front","average":{"1min":35.832000000000001,"5min":7.8520000000000003,"15min":3.1880000000000002},"min":{"1min":0.57199999999999995,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":561.14800000000002,"5min":561.14800000000002,"15min":561.14800000000002},"last":1.1899999999999999}]},{"osd":7,"last update":"Sat Mar 13 21:09:08 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":32.476999999999997,"15min":36.127000000000002},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.141,"5min":706.125,"15min":706.125},"last":1.3360000000000001},{"interface":"front","average":{"1min":0.91100000000000003,"5min":5.2119999999999997,"15min":2.484},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.196,"5min":501.31599999999997,"15min":501.31599999999997},"last":1.363}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":240070656,"data_stored":242342288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":409792512,"data_stored":415422441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":218144768,"data_stored":217318366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":656642048,"data_stored":668240007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":91930624,"data_stored":91242002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":661737472,"data_stored":668240007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":634720256,"data_stored":643216085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":21893120,"data_stored":21713409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":199520256,"data_stored":216598469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":107315200,"data_stored":112042429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47468544,"data_stored":50572939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98734080,"data_stored":107776226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229543936,"data_stored":229219607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":314773504,"data_stored":338041850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":109944832,"data_stored":117177178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196206592,"data_stored":216598469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-14T06:23:25.661 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:10:16.377 DEBUG:teuthology.orchestra.run.gibba004:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 , 2021-03-13T21:10:16.497 INFO:teuthology.orchestra.run.gibba004.stdout:{ , 2021-03-13T21:10:16.497 INFO:teuthology.orchestra.run.gibba004.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-03-14T06:23:25.662 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T21:10:40.265 INFO:tasks.ceph.osd.1:Stopped , 2021-03-13T21:10:40.266 DEBUG:tasks.ceph.osd.2:waiting for process to exit , 2021-03-13T21:10:40.266 INFO:teuthology.orchestra.run:waiting for 300 2021-03-14T06:23:25.662 WARNING:scrape:Ignoring malparsed backtrace: yield vars , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/install/__init__.py", line 619, in task , yield 2021-03-14T06:23:25.663 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 179, in _raise_for_status 2021-03-14T06:23:28.406 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3ef0805308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f3ef0805522] 2021-03-14T06:23:28.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.751 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 4294967295.9999762 , 2021-03-13T23:58:43.751 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:43.751 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.772 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.053504168999999997 , 2021-03-13T23:58:43.772 INFO:teuthology.orchestra.run.gibba008.stdout: } , 2021-03-13T23:58:43.773 INFO:teuthology.orchestra.run.gibba008.stdout: ] 2021-03-14T06:23:28.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.794 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:11.942629+0000", , 2021-03-13T23:58:43.794 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 5.0813000000000003e-05 , 2021-03-13T23:58:43.794 INFO:teuthology.orchestra.run.gibba008.stdout: }, 2021-03-14T06:23:28.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.816 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "header_read", , 2021-03-13T23:58:43.816 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:12.067179+0000", , 2021-03-13T23:58:43.816 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.001232917 2021-03-14T06:23:28.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.837 INFO:teuthology.orchestra.run.gibba008.stdout: "age": 31.354982940999999, , 2021-03-13T23:58:43.837 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.086728439000000004, , 2021-03-13T23:58:43.837 INFO:teuthology.orchestra.run.gibba008.stdout: "type_data": { 2021-03-14T06:23:28.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.859 INFO:teuthology.orchestra.run.gibba008.stdout: { , 2021-03-13T23:58:43.859 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "commit_sent", , 2021-03-13T23:58:43.859 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:17.146212+0000", 2021-03-14T06:23:28.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.880 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:43.880 INFO:teuthology.orchestra.run.gibba008.stdout: { , 2021-03-13T23:58:43.880 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "dispatched", 2021-03-14T06:23:28.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.901 INFO:teuthology.orchestra.run.gibba008.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T23:58:43.901 INFO:teuthology.orchestra.run.gibba008.stdout: "events": [ , 2021-03-13T23:58:43.902 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.922 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.058029955000000001 , 2021-03-13T23:58:43.922 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:43.922 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.943 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:26.004131+0000", , 2021-03-13T23:58:43.943 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.69672588999999996 , 2021-03-13T23:58:43.943 INFO:teuthology.orchestra.run.gibba008.stdout: }, 2021-03-14T06:23:28.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.964 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.053219584, , 2021-03-13T23:58:43.964 INFO:teuthology.orchestra.run.gibba008.stdout: "type_data": { , 2021-03-13T23:58:43.964 INFO:teuthology.orchestra.run.gibba008.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-14T06:23:28.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:43.985 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "started", , 2021-03-13T23:58:43.985 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:38.303313+0000", , 2021-03-13T23:58:43.985 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 0.045620508999999997 2021-03-14T06:23:28.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.363 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.363 INFO:teuthology.orchestra.run.gibba008.stdout: { , 2021-03-13T23:58:44.363 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "done", 2021-03-14T06:23:28.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.384 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 5.3848999999999999e-05 , 2021-03-13T23:58:44.384 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.385 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.405 INFO:teuthology.orchestra.run.gibba008.stdout: "tid": 4087 , 2021-03-13T23:58:44.405 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.405 INFO:teuthology.orchestra.run.gibba008.stdout: "events": [ 2021-03-14T06:23:28.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.427 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:17.161480+0000", , 2021-03-13T23:58:44.427 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 2.035e-06 , 2021-03-13T23:58:44.428 INFO:teuthology.orchestra.run.gibba008.stdout: }, 2021-03-14T06:23:28.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.450 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "all_read", , 2021-03-13T23:58:44.450 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:16.865499+0000", , 2021-03-13T23:58:44.451 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 4.0620000000000002e-06 2021-03-14T06:23:28.418 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.475 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "started", , 2021-03-13T23:58:44.475 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:24.964721+0000", , 2021-03-13T23:58:44.476 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 8.7824000000000005e-05 2021-03-14T06:23:28.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.499 INFO:teuthology.orchestra.run.gibba008.stdout: } , 2021-03-13T23:58:44.499 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.499 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.522 INFO:teuthology.orchestra.run.gibba008.stdout: "client": "client.4568", , 2021-03-13T23:58:44.522 INFO:teuthology.orchestra.run.gibba008.stdout: "client_addr": "v1:172.21.2.108:0/804484500", , 2021-03-13T23:58:44.522 INFO:teuthology.orchestra.run.gibba008.stdout: "tid": 4965 2021-03-14T06:23:28.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.546 INFO:teuthology.orchestra.run.gibba008.stdout: { , 2021-03-13T23:58:44.547 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "dispatched", , 2021-03-13T23:58:44.547 INFO:teuthology.orchestra.run.gibba008.stdout: "time": "2021-03-13T23:58:26.605415+0000", 2021-03-14T06:23:28.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.572 INFO:teuthology.orchestra.run.gibba008.stdout: "client_info": { , 2021-03-13T23:58:44.573 INFO:teuthology.orchestra.run.gibba008.stdout: "client": "client.4568", , 2021-03-13T23:58:44.573 INFO:teuthology.orchestra.run.gibba008.stdout: "client_addr": "v1:172.21.2.108:0/804484500", 2021-03-14T06:23:28.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.598 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.598 INFO:teuthology.orchestra.run.gibba008.stdout: { , 2021-03-13T23:58:44.599 INFO:teuthology.orchestra.run.gibba008.stdout: "event": "commit_sent", 2021-03-14T06:23:28.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.622 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 7.0339999999999999e-06 , 2021-03-13T23:58:44.622 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.623 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.643 INFO:teuthology.orchestra.run.gibba008.stdout: "duration": 2.7905000000000001e-05 , 2021-03-13T23:58:44.643 INFO:teuthology.orchestra.run.gibba008.stdout: }, , 2021-03-13T23:58:44.643 INFO:teuthology.orchestra.run.gibba008.stdout: { 2021-03-14T06:23:28.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:44.907 INFO:teuthology.orchestra.run.gibba042.stdout: "complaint_time": 30, , 2021-03-13T23:58:44.907 INFO:teuthology.orchestra.run.gibba042.stdout: "num_blocked_ops": 0 , 2021-03-13T23:58:44.907 INFO:teuthology.orchestra.run.gibba042.stdout:} 2021-03-14T06:23:28.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.076 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:57:56.573817+0000", , 2021-03-13T23:58:45.076 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 4.6970000000000001e-06 , 2021-03-13T23:58:45.076 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.097 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:02.249523+0000", , 2021-03-13T23:58:45.098 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.014612635000000001 , 2021-03-13T23:58:45.098 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.119 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:02.274660+0000", , 2021-03-13T23:58:45.120 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.000102928 , 2021-03-13T23:58:45.120 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.141 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:02.257906+0000", , 2021-03-13T23:58:45.141 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 3.8411999999999997e-05 , 2021-03-13T23:58:45.142 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.163 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:01.233108+0000", , 2021-03-13T23:58:45.163 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 4.9840000000000001e-06 , 2021-03-13T23:58:45.164 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.185 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:00.940276+0000", , 2021-03-13T23:58:45.185 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 2.886e-06 , 2021-03-13T23:58:45.185 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.207 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:00.940521+0000", , 2021-03-13T23:58:45.207 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.29264311900000001 , 2021-03-13T23:58:45.207 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.229 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:02.258317+0000", , 2021-03-13T23:58:45.229 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.006288353 , 2021-03-13T23:58:45.229 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:28.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.251 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 1.26055027, , 2021-03-13T23:58:45.251 INFO:teuthology.orchestra.run.gibba042.stdout: "type_data": { , 2021-03-13T23:58:45.251 INFO:teuthology.orchestra.run.gibba042.stdout: "flag_point": "reached pg", 2021-03-14T06:23:28.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.273 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "all_read", , 2021-03-13T23:58:45.273 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:01.004807+0000", , 2021-03-13T23:58:45.273 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 1.1729000000000001e-05 2021-03-14T06:23:28.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.294 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "waiting for peered", , 2021-03-13T23:58:45.295 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:01.004983+0000", , 2021-03-13T23:58:45.295 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.22828206100000001 2021-03-14T06:23:28.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.316 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "started", , 2021-03-13T23:58:45.316 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:02.258714+0000", , 2021-03-13T23:58:45.317 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.0033683519999999998 2021-03-14T06:23:28.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.338 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T23:58:45.338 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "throttled", , 2021-03-13T23:58:45.338 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:01.151257+0000", 2021-03-14T06:23:28.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.360 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T23:58:45.360 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "reached_pg", , 2021-03-13T23:58:45.360 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:31.228269+0000", 2021-03-14T06:23:28.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.382 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T23:58:45.382 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "waiting for rw locks", , 2021-03-13T23:58:45.382 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T23:58:39.129741+0000", 2021-03-14T06:23:28.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.403 INFO:teuthology.orchestra.run.gibba042.stdout: }, , 2021-03-13T23:58:45.403 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T23:58:45.404 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "header_read", 2021-03-14T06:23:28.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:58:45.707 INFO:teuthology.orchestra.run.gibba042.stdout:ERROR: (22) Invalid argument , 2021-03-13T23:58:45.707 INFO:teuthology.orchestra.run.gibba042.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. , 2021-03-13T23:58:45.714 DEBUG:teuthology.orchestra.run.gibba042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2021-03-14T06:23:28.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:59:07.507 INFO:teuthology.orchestra.run.gibba008.stdout:55834574872 , 2021-03-13T23:59:07.508 DEBUG:teuthology.orchestra.run.gibba008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats , 2021-03-13T23:59:07.639 INFO:teuthology.orchestra.run.gibba008.stdout:60129542170 2021-03-14T06:23:28.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:59:16.602 DEBUG:teuthology.orchestra.run.gibba008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 , 2021-03-13T23:59:16.724 INFO:teuthology.orchestra.run.gibba008.stdout:{ , 2021-03-13T23:59:16.725 INFO:teuthology.orchestra.run.gibba008.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-03-14T06:23:28.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T23:59:58.531 INFO:tasks.ceph.osd.4:Stopped , 2021-03-13T23:59:58.532 DEBUG:tasks.ceph.osd.5:waiting for process to exit , 2021-03-13T23:59:58.532 INFO:teuthology.orchestra.run:waiting for 300 2021-03-14T06:23:28.438 WARNING:scrape:Ignoring malparsed backtrace: File "src/gevent/greenlet.py", line 829, in gevent._gevent_cgreenlet.Greenlet.get , File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2021-03-14T06:23:28.439 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ 2021-03-14T06:23:29.420 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f59ed533308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f59ed533522] 2021-03-14T06:23:29.421 WARNING:scrape:Ignoring malparsed backtrace: proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait , self._raise_for_status() 2021-03-14T06:23:29.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T12:37:36.783 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet , 2021-03-13T12:37:36.785 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet , 2021-03-13T12:37:36.786 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2021-03-14T06:23:29.425 WARNING:scrape:Ignoring malparsed backtrace: instructed osd(s) 2 to deep-scrub , 2021-03-13T12:37:46.183 INFO:tasks.ceph:Scrubbing osd.3 , 2021-03-13T12:37:46.184 DEBUG:teuthology.orchestra.run.gibba009:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2021-03-14T06:23:29.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T12:37:48.263 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2021-03-13T12:37:38.456852+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=37, tm_sec=38, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=37, tm_sec=44, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T12:37:48.263 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2021-03-13T12:36:46.502277+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=36, tm_sec=46, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=37, tm_sec=44, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T12:37:48.266 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2021-03-13T12:33:17.314950+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=33, tm_sec=17, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=12, tm_min=37, tm_sec=44, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:29.429 WARNING:scrape:Ignoring malparsed backtrace: File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ , next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 277, in task 2021-03-14T06:23:29.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T12:38:51.936 DEBUG:teuthology.orchestra.run.gibba009:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.tWay6n064Z , 2021-03-13T12:38:52.227 DEBUG:teuthology.orchestra.run.gibba009:> sudo chmod 0666 /tmp/tmp.tWay6n064Z , 2021-03-13T12:38:52.300 DEBUG:teuthology.orchestra.remote:gibba009:/tmp/tmp.tWay6n064Z is 2MB 2021-03-14T06:23:29.430 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T12:39:39.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 , 2021-03-13T12:39:39.776 DEBUG:teuthology.run_tasks:Unwinding manager pcp , 2021-03-13T12:39:39.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-03-14T06:23:29.430 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) 2021-03-14T06:23:30.802 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f2d889a0308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f2d889a0522] 2021-03-14T06:23:30.803 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:45.995 INFO:teuthology.orchestra.run.gibba003.stdout: "description": "osd_op(client.4403.0:7248 3.7s1 3:e0563ae6:::gibba00336065-687:head [rollback 86] snapc 9b=[9b,9a,98,95,93,8e,8b,89,88,87,86,81,7c,7a,79,76,74,72,6c,69,5e,5d,5a,58,4f,3e] ondisk+write+known_if_redirected e186)", , 2021-03-13T19:35:45.996 INFO:teuthology.orchestra.run.gibba003.stdout: "initiated_at": "2021-03-13T19:30:21.632942+0000", , 2021-03-13T19:35:45.996 INFO:teuthology.orchestra.run.gibba003.stdout: "age": 324.35774163100001, 2021-03-14T06:23:30.804 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.068 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.068 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:46.068 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "done", 2021-03-14T06:23:30.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.148 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.149 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:46.149 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "commit_sent", 2021-03-14T06:23:30.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.222 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.223 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:46.223 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "done", 2021-03-14T06:23:30.806 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.272 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.272 INFO:teuthology.orchestra.run.gibba003.stdout: "events": [ , 2021-03-13T19:35:46.272 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.807 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.295 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 5.1880000000000001e-06 , 2021-03-13T19:35:46.295 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.296 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.317 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 5.7400000000000001e-06 , 2021-03-13T19:35:46.317 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.317 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.344 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.0024966110000000001 , 2021-03-13T19:35:46.362 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.362 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.809 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.384 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 1.37e-06 , 2021-03-13T19:35:46.384 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.384 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.405 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 1.012756921 , 2021-03-13T19:35:46.406 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.406 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.428 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 7.6416999999999997e-05 , 2021-03-13T19:35:46.428 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.428 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.450 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.019816139999999999 , 2021-03-13T19:35:46.450 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.450 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.812 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.472 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.002598141 , 2021-03-13T19:35:46.472 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.472 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.493 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.0012378949999999999 , 2021-03-13T19:35:46.493 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:46.494 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.515 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:19.428666+0000", , 2021-03-13T19:35:46.515 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 2.7010000000000001e-05 , 2021-03-13T19:35:46.515 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.814 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.537 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:19.428826+0000", , 2021-03-13T19:35:46.537 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 1.0591561650000001 , 2021-03-13T19:35:46.538 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.815 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.558 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:20.549357+0000", , 2021-03-13T19:35:46.559 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.000119764 , 2021-03-13T19:35:46.559 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.580 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:20.549865+0000", , 2021-03-13T19:35:46.580 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.00014975000000000001 , 2021-03-13T19:35:46.581 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:46.601 INFO:teuthology.orchestra.run.gibba003.stdout: } , 2021-03-13T19:35:46.602 INFO:teuthology.orchestra.run.gibba003.stdout: ] , 2021-03-13T19:35:46.602 INFO:teuthology.orchestra.run.gibba003.stdout:} 2021-03-14T06:23:30.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.001 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.002 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", , 2021-03-13T19:35:47.002 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:32:40.378252+0000", 2021-03-14T06:23:30.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.070 INFO:teuthology.orchestra.run.gibba003.stdout: "initiated_at": "2021-03-13T19:32:40.378281+0000", , 2021-03-13T19:35:47.071 INFO:teuthology.orchestra.run.gibba003.stdout: "age": 186.59323788699999, , 2021-03-13T19:35:47.071 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 5.9196846570000003, 2021-03-14T06:23:30.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.114 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.114 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.114 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "dispatched", 2021-03-14T06:23:30.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.166 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.166 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.167 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "waiting for readable", 2021-03-14T06:23:30.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.194 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.194 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.214 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "sub_op_started", 2021-03-14T06:23:30.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.255 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.255 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.256 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "done", 2021-03-14T06:23:30.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.292 INFO:teuthology.orchestra.run.gibba003.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T19:35:47.292 INFO:teuthology.orchestra.run.gibba003.stdout: "client_info": { , 2021-03-13T19:35:47.292 INFO:teuthology.orchestra.run.gibba003.stdout: "client": "client.4403", 2021-03-14T06:23:30.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.346 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 7.6277000000000002e-05 , 2021-03-13T19:35:47.347 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.347 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.398 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 2.2579999999999998e-06 , 2021-03-13T19:35:47.399 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.399 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.451 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.98651681300000005 , 2021-03-13T19:35:47.451 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.451 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.824 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.495 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.032434886000000003 , 2021-03-13T19:35:47.495 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.496 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.825 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.533 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 6.4992000000000004e-05 , 2021-03-13T19:35:47.533 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.533 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.825 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.578 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 1.021492488 , 2021-03-13T19:35:47.578 INFO:teuthology.orchestra.run.gibba003.stdout: } , 2021-03-13T19:35:47.578 INFO:teuthology.orchestra.run.gibba003.stdout: ] 2021-03-14T06:23:30.826 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.624 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.624 INFO:teuthology.orchestra.run.gibba003.stdout: "description": "osd_op(client.4403.0:9998 3.0s1 3:05441f56:::gibba00336065-39:head [setxattr _header (57) in=7b] snapc 139=[139,138,136,132,130,12c,12b,129,126,121,11b,117,115,113,10f,10d,10c,10a,109,ff,fd,f8,f7,f5,f0,e6,e3,e2,dc,d7,d6,d5,cf,cb,c1,bf,b8,b7] ondisk+write+known_if_redirected e369)", , 2021-03-13T19:35:47.624 INFO:teuthology.orchestra.run.gibba003.stdout: "initiated_at": "2021-03-13T19:33:32.348537+0000", 2021-03-14T06:23:30.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.659 INFO:teuthology.orchestra.run.gibba003.stdout: "type_data": { , 2021-03-13T19:35:47.659 INFO:teuthology.orchestra.run.gibba003.stdout: "flag_point": "reached pg", , 2021-03-13T19:35:47.659 INFO:teuthology.orchestra.run.gibba003.stdout: "client_info": { 2021-03-14T06:23:30.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.712 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:18.394669+0000", , 2021-03-13T19:35:47.712 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 2.7569e-05 , 2021-03-13T19:35:47.713 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.828 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.750 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:34:18.394773+0000", , 2021-03-13T19:35:47.750 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 2.2840000000000001e-06 , 2021-03-13T19:35:47.766 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.829 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.955 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.955 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.955 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "queued_for_pg", 2021-03-14T06:23:30.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:47.996 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:47.996 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:47.996 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "waiting for peered", 2021-03-14T06:23:30.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.252 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:48.252 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "sub_op_committed", , 2021-03-13T19:35:48.253 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:29:32.233637+0000", 2021-03-14T06:23:30.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.389 INFO:teuthology.orchestra.run.gibba003.stdout: ] , 2021-03-13T19:35:48.389 INFO:teuthology.orchestra.run.gibba003.stdout: } , 2021-03-13T19:35:48.389 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.427 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.428 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:48.428 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "throttled", 2021-03-14T06:23:30.832 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.453 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.453 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:48.453 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", 2021-03-14T06:23:30.833 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.475 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.476 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:48.476 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "done", 2021-03-14T06:23:30.834 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.500 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0 , 2021-03-13T19:35:48.501 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.501 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.528 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 1.6344000000000001e-05 , 2021-03-13T19:35:48.528 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.528 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.835 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.553 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.010567193000000001 , 2021-03-13T19:35:48.553 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.554 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.836 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.576 INFO:teuthology.orchestra.run.gibba003.stdout: "type_data": { , 2021-03-13T19:35:48.576 INFO:teuthology.orchestra.run.gibba003.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T19:35:48.576 INFO:teuthology.orchestra.run.gibba003.stdout: "client_info": { 2021-03-14T06:23:30.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.616 INFO:teuthology.orchestra.run.gibba003.stdout: "tid": 11852 , 2021-03-13T19:35:48.616 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:48.616 INFO:teuthology.orchestra.run.gibba003.stdout: "events": [ 2021-03-14T06:23:30.837 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.641 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:36.336568+0000", , 2021-03-13T19:35:48.641 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0 , 2021-03-13T19:35:48.641 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.838 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.663 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:36.336718+0000", , 2021-03-13T19:35:48.663 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 3.8785000000000002e-05 , 2021-03-13T19:35:48.664 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.839 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.686 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:36.336838+0000", , 2021-03-13T19:35:48.687 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 3.5389999999999999e-06 , 2021-03-13T19:35:48.687 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.708 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:37.474535+0000", , 2021-03-13T19:35:48.709 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 3.3069999999999998e-06 , 2021-03-13T19:35:48.709 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.840 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.730 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 5.9152403280000003, , 2021-03-13T19:35:48.730 INFO:teuthology.orchestra.run.gibba003.stdout: "type_data": { , 2021-03-13T19:35:48.731 INFO:teuthology.orchestra.run.gibba003.stdout: "flag_point": "reached pg", 2021-03-14T06:23:30.841 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:48.752 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "queued_for_pg", , 2021-03-13T19:35:48.752 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:41.407934+0000", , 2021-03-13T19:35:48.753 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 4.1124000000000001e-05 2021-03-14T06:23:30.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.086 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:49.087 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "dispatched", , 2021-03-13T19:35:49.087 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:28:04.199502+0000", 2021-03-14T06:23:30.842 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.140 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:49.140 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:49.141 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "dispatched", 2021-03-14T06:23:30.843 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.177 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.439463671 , 2021-03-13T19:35:49.177 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:49.178 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.215 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:28:50.800623+0000", , 2021-03-13T19:35:49.215 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 8.4149999999999999e-06 , 2021-03-13T19:35:49.216 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.844 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.254 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:28:50.802203+0000", , 2021-03-13T19:35:49.254 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.450967917 , 2021-03-13T19:35:49.254 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.845 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.311 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "header_read", , 2021-03-13T19:35:49.311 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:28:50.802133+0000", , 2021-03-13T19:35:49.311 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 2.993e-06 2021-03-14T06:23:30.846 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.351 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "waiting for rw locks", , 2021-03-13T19:35:49.352 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:28:50.802588+0000", , 2021-03-13T19:35:49.352 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.45300842299999999 2021-03-14T06:23:30.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.408 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:49.408 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "header_read", , 2021-03-13T19:35:49.409 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:29:50.504135+0000", 2021-03-14T06:23:30.847 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.456 INFO:teuthology.orchestra.run.gibba011.stdout: "events": [ , 2021-03-13T19:35:49.456 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:49.456 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "initiated", 2021-03-14T06:23:30.848 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.512 INFO:teuthology.orchestra.run.gibba011.stdout: "description": "MOSDECSubOpWriteReply(3.cs0 498/497 ECSubWriteReply(tid=2253, last_complete=498'2920, committed=1, applied=1))", , 2021-03-13T19:35:49.512 INFO:teuthology.orchestra.run.gibba011.stdout: "initiated_at": "2021-03-13T19:35:41.668663+0000", , 2021-03-13T19:35:49.512 INFO:teuthology.orchestra.run.gibba011.stdout: "age": 7.4090808729999997, 2021-03-14T06:23:30.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.574 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 2.3501000000000001e-05 , 2021-03-13T19:35:49.574 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:49.574 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.849 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:49.625 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:35:47.819751+0000", , 2021-03-13T19:35:49.625 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 4.6920000000000001e-06 , 2021-03-13T19:35:49.626 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.850 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.004 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.005 INFO:teuthology.orchestra.run.gibba011.stdout: "description": "osd_op(client.4403.0:8987 3.6s0 3:64eacf87:::gibba00336065-767:head [copy-from ver 1176] snapc f7=[f7,f6,f5,f4,f1,f0,ed,ec,e9,e7,e6,e5,e4,e3,e2,e0,dc,da,d7,d6,d5,cf,cb,c8,c6,c2,c1,bf,b8,b7,af,a5,8e,86,74,72] ondisk+write+known_if_redirected e300)", , 2021-03-13T19:35:50.005 INFO:teuthology.orchestra.run.gibba011.stdout: "initiated_at": "2021-03-13T19:32:23.343972+0000", 2021-03-14T06:23:30.851 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.064 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:32:23.344066+0000", , 2021-03-13T19:35:50.064 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 2.5000000000000002e-06 , 2021-03-13T19:35:50.064 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.105 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:48.277657+0000", , 2021-03-13T19:35:50.105 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.6339999999999999e-06 , 2021-03-13T19:35:50.105 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.852 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.157 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:49.274225+0000", , 2021-03-13T19:35:50.158 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.0079159920000000002 , 2021-03-13T19:35:50.158 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.853 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.208 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:49.283233+0000", , 2021-03-13T19:35:50.208 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 5.3372999999999999e-05 , 2021-03-13T19:35:50.208 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.251 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 4.8051721269999996, , 2021-03-13T19:35:50.251 INFO:teuthology.orchestra.run.gibba011.stdout: "type_data": { , 2021-03-13T19:35:50.252 INFO:teuthology.orchestra.run.gibba011.stdout: "flag_point": "started", 2021-03-14T06:23:30.854 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.308 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "header_read", , 2021-03-13T19:35:50.308 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:58.653359+0000", , 2021-03-13T19:35:50.308 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 2.463e-06 2021-03-14T06:23:30.855 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.380 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "reached_pg", , 2021-03-13T19:35:50.381 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:03.445749+0000", , 2021-03-13T19:35:50.381 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 4.5599000000000002e-05 2021-03-14T06:23:30.856 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.408 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "commit_sent", , 2021-03-13T19:35:50.409 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:20.553429+0000", , 2021-03-13T19:35:50.409 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.000228211 2021-03-14T06:23:30.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.460 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "done", , 2021-03-13T19:35:50.460 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:20.614488+0000", , 2021-03-13T19:35:50.460 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.1409738890000001 2021-03-14T06:23:30.857 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.565 INFO:teuthology.orchestra.run.gibba011.stdout: } , 2021-03-13T19:35:50.566 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:50.566 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.858 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.662 INFO:teuthology.orchestra.run.gibba011.stdout: "age": 90.525046462000006, , 2021-03-13T19:35:50.663 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.145254129, , 2021-03-13T19:35:50.669 INFO:teuthology.orchestra.run.gibba011.stdout: "type_data": { 2021-03-14T06:23:30.859 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.787 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.788 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "initiated", , 2021-03-13T19:35:50.788 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.492450+0000", 2021-03-14T06:23:30.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.855 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.855 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "throttled", , 2021-03-13T19:35:50.856 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.494670+0000", 2021-03-14T06:23:30.860 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.897 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.897 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "header_read", , 2021-03-13T19:35:50.897 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.495756+0000", 2021-03-14T06:23:30.861 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.920 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.920 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "all_read", , 2021-03-13T19:35:50.920 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.495813+0000", 2021-03-14T06:23:30.862 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:50.945 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:50.945 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "dispatched", , 2021-03-13T19:35:50.945 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.495871+0000", 2021-03-14T06:23:30.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.024 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.024 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "reached_pg", , 2021-03-13T19:35:51.024 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:35:32.551090+0000", 2021-03-14T06:23:30.863 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.440 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.8312e-05 , 2021-03-13T19:35:51.440 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:51.440 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.864 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.472 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:31:26.606156+0000", , 2021-03-13T19:35:51.472 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 2.2018000000000002e-05 , 2021-03-13T19:35:51.472 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.865 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.496 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "initiated", , 2021-03-13T19:35:51.496 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:31:30.614179+0000", , 2021-03-13T19:35:51.496 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0 2021-03-14T06:23:30.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.522 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "queued_for_pg", , 2021-03-13T19:35:51.523 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:31:30.614517+0000", , 2021-03-13T19:35:51.523 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.00028579000000000001 2021-03-14T06:23:30.866 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.576 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "done", , 2021-03-13T19:35:51.576 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:31:31.443009+0000", , 2021-03-13T19:35:51.576 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.82847447299999999 2021-03-14T06:23:30.867 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.657 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.657 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "done", , 2021-03-13T19:35:51.658 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:31:36.316487+0000", 2021-03-14T06:23:30.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.721 INFO:teuthology.orchestra.run.gibba011.stdout: "initiated_at": "2021-03-13T19:31:58.650727+0000", , 2021-03-13T19:35:51.721 INFO:teuthology.orchestra.run.gibba011.stdout: "age": 232.77401456000001, , 2021-03-13T19:35:51.721 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 9.2828313930000004, 2021-03-14T06:23:30.868 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.805 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:51.806 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.806 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "throttled", 2021-03-14T06:23:30.869 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.894 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:51.895 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.896 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "all_read", 2021-03-14T06:23:30.870 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.940 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:51.940 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.942 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "reached_pg", 2021-03-14T06:23:30.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:51.973 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:51.973 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:51.975 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "reached_pg", 2021-03-14T06:23:30.871 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.005 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:52.006 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:52.006 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "sub_op_started", 2021-03-14T06:23:30.872 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.034 INFO:teuthology.orchestra.run.gibba011.stdout: "description": "osd_op(client.4403.0:9463 3.7s0 3:e77e4fcc:::gibba00336065-333:head [append 0~400000] snapc 110=[110,10f,10d,10c,10b,10a,109,ff,fd,f8,f7,f6,f5,f1,f0,ed,ec,e7,e6,e5,e3,e2,dc,d7,d6,d5,cf,cb,c8,c6,c2,c1,bf,b8,b7,a5,86,74] ondisk+write+known_if_redirected e326)", , 2021-03-13T19:35:52.035 INFO:teuthology.orchestra.run.gibba011.stdout: "initiated_at": "2021-03-13T19:32:49.463901+0000", , 2021-03-13T19:35:52.035 INFO:teuthology.orchestra.run.gibba011.stdout: "age": 181.960839706, 2021-03-14T06:23:30.873 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.115 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0 , 2021-03-13T19:35:52.115 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:52.115 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.206 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.2969999999999999e-05 , 2021-03-13T19:35:52.206 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:52.207 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.874 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.371 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", , 2021-03-13T19:35:52.371 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:42.303343+0000", , 2021-03-13T19:35:52.375 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.00012329499999999999 2021-03-14T06:23:30.875 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.698 INFO:teuthology.orchestra.run.gibba003.stdout: "events": [ , 2021-03-13T19:35:52.699 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.699 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "initiated", 2021-03-14T06:23:30.876 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.754 INFO:teuthology.orchestra.run.gibba003.stdout: "events": [ , 2021-03-13T19:35:52.754 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.754 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "initiated", 2021-03-14T06:23:30.877 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.826 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:52.826 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.826 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "header_read", 2021-03-14T06:23:30.877 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.863 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:52.864 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.864 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", 2021-03-14T06:23:30.878 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.917 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:52.917 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.917 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", 2021-03-14T06:23:30.879 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:52.963 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:52.963 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:52.963 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "sub_op_started", 2021-03-14T06:23:30.879 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.029 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.029 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:53.030 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "waiting for rw locks", 2021-03-14T06:23:30.880 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.073 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.073 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:53.073 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "reached_pg", 2021-03-14T06:23:30.881 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.127 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.127 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:53.127 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "waiting for peered", 2021-03-14T06:23:30.882 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.197 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.197 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:53.198 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "dispatched", 2021-03-14T06:23:30.882 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.267 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.267 INFO:teuthology.orchestra.run.gibba003.stdout: { , 2021-03-13T19:35:53.267 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "all_read", 2021-03-14T06:23:30.883 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.312 INFO:teuthology.orchestra.run.gibba003.stdout: } , 2021-03-13T19:35:53.312 INFO:teuthology.orchestra.run.gibba003.stdout: ] , 2021-03-13T19:35:53.312 INFO:teuthology.orchestra.run.gibba003.stdout: } 2021-03-14T06:23:30.884 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.398 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 4.7380000000000002e-06 , 2021-03-13T19:35:53.398 INFO:teuthology.orchestra.run.gibba003.stdout: }, , 2021-03-13T19:35:53.399 INFO:teuthology.orchestra.run.gibba003.stdout: { 2021-03-14T06:23:30.885 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.449 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:17.460689+0000", , 2021-03-13T19:35:53.449 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 3.6547000000000001e-05 , 2021-03-13T19:35:53.449 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.885 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.523 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:33.589615+0000", , 2021-03-13T19:35:53.523 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 0.00014986099999999999 , 2021-03-13T19:35:53.524 INFO:teuthology.orchestra.run.gibba003.stdout: }, 2021-03-14T06:23:30.886 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.585 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:37.330030+0000", , 2021-03-13T19:35:53.585 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 6.0004359139999996 , 2021-03-13T19:35:53.586 INFO:teuthology.orchestra.run.gibba003.stdout: } 2021-03-14T06:23:30.887 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.631 INFO:teuthology.orchestra.run.gibba003.stdout: "event": "queued_for_pg", , 2021-03-13T19:35:53.632 INFO:teuthology.orchestra.run.gibba003.stdout: "time": "2021-03-13T19:35:47.319040+0000", , 2021-03-13T19:35:53.632 INFO:teuthology.orchestra.run.gibba003.stdout: "duration": 3.0301999999999999e-05 2021-03-14T06:23:30.888 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:53.977 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:53.977 INFO:teuthology.orchestra.run.gibba011.stdout: "events": [ , 2021-03-13T19:35:53.978 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.888 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.045 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 8.3610000000000001e-06 , 2021-03-13T19:35:54.045 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.046 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.889 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.135 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 1.6915e-05 , 2021-03-13T19:35:54.135 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.135 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.230 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 10.117428668000001 , 2021-03-13T19:35:54.231 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.231 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.289 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.0096703420000000002 , 2021-03-13T19:35:54.290 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.290 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.421 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 5.0018000000000001e-05 , 2021-03-13T19:35:54.422 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.422 INFO:teuthology.orchestra.run.gibba011.stdout: { 2021-03-14T06:23:30.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.508 INFO:teuthology.orchestra.run.gibba011.stdout: "type_data": { , 2021-03-13T19:35:54.529 INFO:teuthology.orchestra.run.gibba011.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T19:35:54.529 INFO:teuthology.orchestra.run.gibba011.stdout: "client_info": { 2021-03-14T06:23:30.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.594 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:54.439935+0000", , 2021-03-13T19:35:54.594 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 3.9758000000000001e-05 , 2021-03-13T19:35:54.594 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.683 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:33:54.440070+0000", , 2021-03-13T19:35:54.684 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 3.6330000000000002e-06 , 2021-03-13T19:35:54.684 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.722 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:20.471874+0000", , 2021-03-13T19:35:54.723 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.00016089000000000001 , 2021-03-13T19:35:54.723 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.895 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.808 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:20.493088+0000", , 2021-03-13T19:35:54.808 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.0050426619999999998 , 2021-03-13T19:35:54.808 INFO:teuthology.orchestra.run.gibba011.stdout: }, 2021-03-14T06:23:30.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.856 INFO:teuthology.orchestra.run.gibba011.stdout: }, , 2021-03-13T19:35:54.857 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:54.857 INFO:teuthology.orchestra.run.gibba011.stdout: "description": "osd_op(client.4403.0:10685 3.3s0 3:c94c3524:::gibba00336065-647:head [setxattr _header (58) in=7b] snapc 166=[166,165,164,161,15f,15e,15d,15c,15b,15a,154,151,150,148,143,13e,13a,138,132,12b,129,126,117,115,10a,f8,f0,e6,e3,dc,d7,d5,cf,cb,c1,bf] ondisk+write+known_if_redirected e416)", 2021-03-14T06:23:30.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.925 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "initiated", , 2021-03-13T19:35:54.925 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:19.429203+0000", , 2021-03-13T19:35:54.926 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0 2021-03-14T06:23:30.897 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:54.957 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "queued_for_pg", , 2021-03-13T19:35:54.957 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:20.226542+0000", , 2021-03-13T19:35:54.957 INFO:teuthology.orchestra.run.gibba011.stdout: "duration": 0.00010336999999999999 2021-03-14T06:23:30.898 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:55.020 INFO:teuthology.orchestra.run.gibba011.stdout: { , 2021-03-13T19:35:55.020 INFO:teuthology.orchestra.run.gibba011.stdout: "event": "all_read", , 2021-03-13T19:35:55.020 INFO:teuthology.orchestra.run.gibba011.stdout: "time": "2021-03-13T19:34:38.515393+0000", 2021-03-14T06:23:30.899 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:35:59.642+0000 7f86845f1700 -1 WARNING: all dangerous and experimental features are enabled. , reweighted osd.7 to 1 (10000) , 2021-03-13T19:36:00.539 DEBUG:teuthology.orchestra.run.gibba003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2021-03-14T06:23:30.901 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:36:11.151 INFO:teuthology.orchestra.run.gibba003.stdout:55834574959 , 2021-03-13T19:36:11.161 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574959 got 55834574959 for osd.3 , 2021-03-13T19:36:11.162 DEBUG:teuthology.orchestra.run.gibba003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2021-03-14T06:23:30.903 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:36:15.967 INFO:tasks.ceph:Scrubbing osd.4 , 2021-03-13T19:36:15.968 DEBUG:teuthology.orchestra.run.gibba003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 , 2021-03-13T19:36:16.058+0000 7fc5003e5700 -1 WARNING: all dangerous and experimental features are enabled. 2021-03-14T06:23:30.904 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:36:37.909 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2021-03-13T19:36:10.685729+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=36, tm_sec=10, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=36, tm_sec=13, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T19:36:37.909 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2021-03-13T19:36:12.144942+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=36, tm_sec=12, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=36, tm_sec=13, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T19:36:37.910 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2021-03-13T19:35:18.455743+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=35, tm_sec=18, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=19, tm_min=36, tm_sec=13, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:30.904 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:36:59.936 DEBUG:teuthology.orchestra.run.gibba003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 , 2021-03-13T19:37:00.001 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@gibba003.front.sepia.ceph.com , 2021-03-13T19:37:00.002 DEBUG:teuthology.orchestra.run.gibba003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2021-03-14T06:23:30.905 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T19:40:59.546 INFO:teuthology.orchestra.run.gibba011.stdout:^* hv02.front.sepia.ceph.com 2 6 377 12 +867us[ +918us] +/- 42ms , 2021-03-13T19:40:59.546 INFO:teuthology.orchestra.run.gibba011.stdout:^- hv03.front.sepia.ceph.com 3 6 377 13 -1811us[-1761us] +/- 62ms , 2021-03-13T19:40:59.546 INFO:teuthology.orchestra.run.gibba011.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 15 +568us[ +618us] +/- 46ms 2021-03-14T06:23:30.906 WARNING:scrape:Ignoring malparsed backtrace: run.wait(tests.values()) , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() 2021-03-14T06:23:31.373 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f3fc662b308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f3fc662b522] 2021-03-14T06:23:31.374 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.864 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 5.4609999999999999e-06 , 2021-03-13T22:40:29.864 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.864 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.375 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.887 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 2.7611999999999999e-05 , 2021-03-13T22:40:29.887 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.887 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.375 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.910 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 6.6056000000000006e-05 , 2021-03-13T22:40:29.910 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.911 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.376 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.936 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 3.4989999999999999e-06 , 2021-03-13T22:40:29.937 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.937 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.377 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.961 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 6.3099999999999997e-06 , 2021-03-13T22:40:29.961 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.961 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.377 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:29.986 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 2.9960000000000002e-06 , 2021-03-13T22:40:29.986 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:29.987 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.378 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.011 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 5.9747000000000002e-05 , 2021-03-13T22:40:30.011 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:30.011 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.379 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.033 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 1.7458000000000001e-05 , 2021-03-13T22:40:30.033 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:30.033 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.379 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.058 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:30.058 INFO:teuthology.orchestra.run.gibba029.stdout: { , 2021-03-13T22:40:30.058 INFO:teuthology.orchestra.run.gibba029.stdout: "event": "reached_pg", 2021-03-14T06:23:31.380 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.084 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:30.084 INFO:teuthology.orchestra.run.gibba029.stdout: { , 2021-03-13T22:40:30.085 INFO:teuthology.orchestra.run.gibba029.stdout: "event": "waiting for rw locks", 2021-03-14T06:23:31.381 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.110 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 3.6310000000000003e-05 , 2021-03-13T22:40:30.110 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:30.111 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.381 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.144 INFO:teuthology.orchestra.run.gibba029.stdout: "time": "2021-03-13T22:39:01.405765+0000", , 2021-03-13T22:40:30.144 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 3.9926999999999999e-05 , 2021-03-13T22:40:30.145 INFO:teuthology.orchestra.run.gibba029.stdout: }, 2021-03-14T06:23:31.382 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.176 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.062468111, , 2021-03-13T22:40:30.177 INFO:teuthology.orchestra.run.gibba029.stdout: "type_data": { , 2021-03-13T22:40:30.177 INFO:teuthology.orchestra.run.gibba029.stdout: "flag_point": "reached pg", 2021-03-14T06:23:31.383 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.203 INFO:teuthology.orchestra.run.gibba029.stdout: "age": 88.442827965999996, , 2021-03-13T22:40:30.203 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.062851114999999999, , 2021-03-13T22:40:30.203 INFO:teuthology.orchestra.run.gibba029.stdout: "type_data": { 2021-03-14T06:23:31.383 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.226 INFO:teuthology.orchestra.run.gibba029.stdout: { , 2021-03-13T22:40:30.226 INFO:teuthology.orchestra.run.gibba029.stdout: "event": "sub_op_started", , 2021-03-13T22:40:30.226 INFO:teuthology.orchestra.run.gibba029.stdout: "time": "2021-03-13T22:39:01.406463+0000", 2021-03-14T06:23:31.384 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.357 DEBUG:teuthology.orchestra.run.gibba031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops , 2021-03-13T22:40:30.452+0000 7f6260a56700 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 45769) UID: 0 , 2021-03-13T22:40:30.460 INFO:teuthology.orchestra.run.gibba031.stdout:{ 2021-03-14T06:23:31.385 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.630 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.630 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.631 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "reached_pg", 2021-03-14T06:23:31.385 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.659 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 2.0829999999999999e-05 , 2021-03-13T22:40:30.660 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.660 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.386 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.684 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0.018925114999999999 , 2021-03-13T22:40:30.684 INFO:teuthology.orchestra.run.gibba031.stdout: } , 2021-03-13T22:40:30.684 INFO:teuthology.orchestra.run.gibba031.stdout: ] 2021-03-14T06:23:31.386 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.710 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:38:57.394693+0000", , 2021-03-13T22:40:30.711 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 3.1189000000000002e-05 , 2021-03-13T22:40:30.711 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.387 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.735 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "initiated", , 2021-03-13T22:40:30.735 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:11.453146+0000", , 2021-03-13T22:40:30.735 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 2021-03-14T06:23:31.388 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.760 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "done", , 2021-03-13T22:40:30.760 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:11.477910+0000", , 2021-03-13T22:40:30.760 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0.024696617000000001 2021-03-14T06:23:31.388 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.790 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.790 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "throttled", , 2021-03-13T22:40:30.791 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:15.551866+0000", 2021-03-14T06:23:31.389 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.815 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.815 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "reached_pg", , 2021-03-13T22:40:30.816 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:15.564378+0000", 2021-03-14T06:23:31.389 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.841 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.841 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "done", , 2021-03-13T22:40:30.841 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:19.555587+0000", 2021-03-14T06:23:31.390 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.866 INFO:teuthology.orchestra.run.gibba031.stdout: "events": [ , 2021-03-13T22:40:30.867 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.867 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "initiated", 2021-03-14T06:23:31.391 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.892 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.892 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.892 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "all_read", 2021-03-14T06:23:31.392 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.915 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.915 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:30.916 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "queued_for_pg", 2021-03-14T06:23:31.392 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.938 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 , 2021-03-13T22:40:30.939 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.939 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:30.966 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 1.0601999999999999e-05 , 2021-03-13T22:40:30.966 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:30.967 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.393 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.309 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.309 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "header_read", , 2021-03-13T22:40:31.309 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:37:49.352908+0000", 2021-03-14T06:23:31.394 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.333 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.334 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "queued_for_pg", , 2021-03-13T22:40:31.334 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:37:49.356542+0000", 2021-03-14T06:23:31.395 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.357 INFO:teuthology.orchestra.run.gibba031.stdout: "events": [ , 2021-03-13T22:40:31.357 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.357 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "initiated", 2021-03-14T06:23:31.395 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.381 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.381 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.381 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "queued_for_pg", 2021-03-14T06:23:31.396 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.404 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.405 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.405 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "done", 2021-03-14T06:23:31.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.428 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 3.7397999999999997e-05 , 2021-03-13T22:40:31.429 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.429 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.397 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.457 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 7.1899999999999998e-06 , 2021-03-13T22:40:31.457 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.457 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.398 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.480 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 8.5266999999999995e-05 , 2021-03-13T22:40:31.480 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.480 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.399 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.513 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 8.2416999999999994e-05 , 2021-03-13T22:40:31.514 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.514 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.399 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.536 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 1.022763774 , 2021-03-13T22:40:31.537 INFO:teuthology.orchestra.run.gibba031.stdout: } , 2021-03-13T22:40:31.537 INFO:teuthology.orchestra.run.gibba031.stdout: ] 2021-03-14T06:23:31.400 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.571 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:31.571 INFO:teuthology.orchestra.run.gibba031.stdout: "description": "osd_op(client.4527.0:1725 3.es1 3:7090ca4c:::gibba00736106-19:head [append 0~540672] snapc 88=[88,87,82,81,7c,7b,78,77,70,6f,6c,62,61,41] RETRY=1 ondisk+retry+write+known_if_redirected e172)", , 2021-03-13T22:40:31.571 INFO:teuthology.orchestra.run.gibba031.stdout: "initiated_at": "2021-03-13T22:39:28.531920+0000", 2021-03-14T06:23:31.400 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.606 INFO:teuthology.orchestra.run.gibba031.stdout: "type_data": { , 2021-03-13T22:40:31.606 INFO:teuthology.orchestra.run.gibba031.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T22:40:31.606 INFO:teuthology.orchestra.run.gibba031.stdout: "client_info": { 2021-03-14T06:23:31.401 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.630 INFO:teuthology.orchestra.run.gibba031.stdout: "tid": 1727 , 2021-03-13T22:40:31.630 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:31.631 INFO:teuthology.orchestra.run.gibba031.stdout: "events": [ 2021-03-14T06:23:31.402 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.651 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:28.534154+0000", , 2021-03-13T22:40:31.652 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 , 2021-03-13T22:40:31.652 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.403 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.674 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:29.536436+0000", , 2021-03-13T22:40:31.674 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 6.7220000000000002e-06 , 2021-03-13T22:40:31.674 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.403 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.698 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:43.374960+0000", , 2021-03-13T22:40:31.698 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 3.9249999999999997e-06 , 2021-03-13T22:40:31.698 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:31.818 INFO:teuthology.orchestra.run.gibba031.stdout:{ , 2021-03-13T22:40:31.819 INFO:teuthology.orchestra.run.gibba031.stdout: "ops": [], , 2021-03-13T22:40:31.819 INFO:teuthology.orchestra.run.gibba031.stdout: "num_ops": 0 2021-03-14T06:23:31.404 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.052 INFO:teuthology.orchestra.run.gibba031.stdout: "tid": 350 , 2021-03-13T22:40:32.052 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.052 INFO:teuthology.orchestra.run.gibba031.stdout: "events": [ 2021-03-14T06:23:31.405 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.077 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:38:27.431539+0000", , 2021-03-13T22:40:32.078 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 3.3079999999999999e-06 , 2021-03-13T22:40:32.078 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.406 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.102 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:38:32.461492+0000", , 2021-03-13T22:40:32.102 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 9.4744e-05 , 2021-03-13T22:40:32.102 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.125 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "throttled", , 2021-03-13T22:40:32.125 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:23.601081+0000", , 2021-03-13T22:40:32.125 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 2.1974e-05 2021-03-14T06:23:31.407 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.149 INFO:teuthology.orchestra.run.gibba031.stdout: } , 2021-03-13T22:40:32.149 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.149 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.173 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.173 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "queued_for_pg", , 2021-03-13T22:40:32.173 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:39:38.408611+0000", 2021-03-14T06:23:31.408 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.197 INFO:teuthology.orchestra.run.gibba031.stdout: "client_info": { , 2021-03-13T22:40:32.197 INFO:teuthology.orchestra.run.gibba031.stdout: "client": "client.4527", , 2021-03-13T22:40:32.198 INFO:teuthology.orchestra.run.gibba031.stdout: "client_addr": "172.21.2.107:0/746744859", 2021-03-14T06:23:31.409 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.221 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.221 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.222 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "reached_pg", 2021-03-14T06:23:31.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.245 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.245 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.245 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "sub_op_committed", 2021-03-14T06:23:31.410 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.268 INFO:teuthology.orchestra.run.gibba031.stdout: "flag_point": "started", , 2021-03-13T22:40:32.268 INFO:teuthology.orchestra.run.gibba031.stdout: "client_info": { , 2021-03-13T22:40:32.268 INFO:teuthology.orchestra.run.gibba031.stdout: "client": "client.4527", 2021-03-14T06:23:31.411 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.292 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 4.0099999999999997e-06 , 2021-03-13T22:40:32.292 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.292 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.315 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0.0036857740000000002 , 2021-03-13T22:40:32.315 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.316 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.412 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.339 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.340 INFO:teuthology.orchestra.run.gibba031.stdout: "description": "osd_op(client.4527.0:2368 3.0s0 3:05082bf0:::gibba00736106-33:head [rollback 9b] snapc c2=[c2,c1,bf,bd,bc,bb,ba,b6,b4,ae,9b] ondisk+write+known_if_redirected e230)", , 2021-03-13T22:40:32.340 INFO:teuthology.orchestra.run.gibba031.stdout: "initiated_at": "2021-03-13T22:40:26.399691+0000", 2021-03-14T06:23:31.413 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.363 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:26.399746+0000", , 2021-03-13T22:40:32.363 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 , 2021-03-13T22:40:32.363 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.701 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:38:57.403127+0000", , 2021-03-13T22:40:32.702 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 , 2021-03-13T22:40:32.702 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.414 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.726 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.726 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.727 INFO:teuthology.orchestra.run.gibba031.stdout: "description": "osd_op(client.4527.0:2129 3.6s0 3:63e2a069:::gibba00736106-50:head [copy-from ver 56] snapc a8=[a8,a6,9b,6c] ondisk+write+known_if_redirected e205)", 2021-03-14T06:23:31.415 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.752 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "header_read", , 2021-03-13T22:40:32.752 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:01.758448+0000", , 2021-03-13T22:40:32.752 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 4.8169999999999998e-06 2021-03-14T06:23:31.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.781 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "reached_pg", , 2021-03-13T22:40:32.781 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:15.974802+0000", , 2021-03-13T22:40:32.782 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0.00014245600000000001 2021-03-14T06:23:31.416 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.809 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.809 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "header_read", , 2021-03-13T22:40:32.809 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:15.974838+0000", 2021-03-14T06:23:31.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.833 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.833 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.833 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "throttled", 2021-03-14T06:23:31.417 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.854 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0 , 2021-03-13T22:40:32.855 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.855 INFO:teuthology.orchestra.run.gibba031.stdout: { 2021-03-14T06:23:31.419 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.876 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.876 INFO:teuthology.orchestra.run.gibba031.stdout: "description": "osd_op(client.4527.0:2380 3.6s0 3:63e2a069:::gibba00736106-50:head [read 0~10403840,checksum] snapc 0=[] ondisk+read+known_if_redirected e230)", , 2021-03-13T22:40:32.877 INFO:teuthology.orchestra.run.gibba031.stdout: "initiated_at": "2021-03-13T22:40:26.472471+0000", 2021-03-14T06:23:31.420 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.898 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:26.472663+0000", , 2021-03-13T22:40:32.898 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 0.00061971900000000002 , 2021-03-13T22:40:32.898 INFO:teuthology.orchestra.run.gibba031.stdout: }, 2021-03-14T06:23:31.421 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.920 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "dispatched", , 2021-03-13T22:40:32.920 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:26.486784+0000", , 2021-03-13T22:40:32.920 INFO:teuthology.orchestra.run.gibba031.stdout: "duration": 2.9330000000000001e-06 2021-03-14T06:23:31.422 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.941 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.941 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "dispatched", , 2021-03-13T22:40:32.941 INFO:teuthology.orchestra.run.gibba031.stdout: "time": "2021-03-13T22:40:26.491496+0000", 2021-03-14T06:23:31.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:32.962 INFO:teuthology.orchestra.run.gibba031.stdout: }, , 2021-03-13T22:40:32.963 INFO:teuthology.orchestra.run.gibba031.stdout: { , 2021-03-13T22:40:32.963 INFO:teuthology.orchestra.run.gibba031.stdout: "event": "dispatched", 2021-03-14T06:23:31.423 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.300 INFO:teuthology.orchestra.run.gibba029.stdout: { , 2021-03-13T22:40:33.301 INFO:teuthology.orchestra.run.gibba029.stdout: "event": "initiated", , 2021-03-13T22:40:33.301 INFO:teuthology.orchestra.run.gibba029.stdout: "time": "2021-03-13T22:40:15.692623+0000", 2021-03-14T06:23:31.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.344 INFO:teuthology.orchestra.run.gibba029.stdout: "initiated_at": "2021-03-13T22:40:15.714598+0000", , 2021-03-13T22:40:33.344 INFO:teuthology.orchestra.run.gibba029.stdout: "age": 17.556852008, , 2021-03-13T22:40:33.344 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.0079393120000000005, 2021-03-14T06:23:31.424 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.370 INFO:teuthology.orchestra.run.gibba029.stdout: "description": "osd_op(client.4527.0:2263 3.15s0 3:aab3d5ba:::gibba00736106-49:head [read 0~3424256,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e219)", , 2021-03-13T22:40:33.370 INFO:teuthology.orchestra.run.gibba029.stdout: "initiated_at": "2021-03-13T22:40:15.718138+0000", , 2021-03-13T22:40:33.370 INFO:teuthology.orchestra.run.gibba029.stdout: "age": 17.553311867000001, 2021-03-14T06:23:31.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.399 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.00041486700000000002 , 2021-03-13T22:40:33.400 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:33.400 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.425 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.422 INFO:teuthology.orchestra.run.gibba029.stdout: "time": "2021-03-13T22:40:15.719179+0000", , 2021-03-13T22:40:33.422 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 3.5407000000000002e-05 , 2021-03-13T22:40:33.422 INFO:teuthology.orchestra.run.gibba029.stdout: }, 2021-03-14T06:23:31.426 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.445 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:33.445 INFO:teuthology.orchestra.run.gibba029.stdout: { , 2021-03-13T22:40:33.445 INFO:teuthology.orchestra.run.gibba029.stdout: "description": "MOSDECSubOpReadReply(3.15s0 219/216 ECSubReadReply(tid=20, attrs_read=0))", 2021-03-14T06:23:31.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.467 INFO:teuthology.orchestra.run.gibba029.stdout: } , 2021-03-13T22:40:33.467 INFO:teuthology.orchestra.run.gibba029.stdout: }, , 2021-03-13T22:40:33.467 INFO:teuthology.orchestra.run.gibba029.stdout: { 2021-03-14T06:23:31.427 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.488 INFO:teuthology.orchestra.run.gibba029.stdout: ] , 2021-03-13T22:40:33.489 INFO:teuthology.orchestra.run.gibba029.stdout: } , 2021-03-13T22:40:33.489 INFO:teuthology.orchestra.run.gibba029.stdout: }, 2021-03-14T06:23:31.428 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.511 INFO:teuthology.orchestra.run.gibba029.stdout: } , 2021-03-13T22:40:33.511 INFO:teuthology.orchestra.run.gibba029.stdout: ] , 2021-03-13T22:40:33.511 INFO:teuthology.orchestra.run.gibba029.stdout: } 2021-03-14T06:23:31.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.534 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.088715450000000001 , 2021-03-13T22:40:33.534 INFO:teuthology.orchestra.run.gibba029.stdout: } , 2021-03-13T22:40:33.534 INFO:teuthology.orchestra.run.gibba029.stdout: ] 2021-03-14T06:23:31.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:33.563 INFO:teuthology.orchestra.run.gibba029.stdout: "time": "2021-03-13T22:40:25.407624+0000", , 2021-03-13T22:40:33.564 INFO:teuthology.orchestra.run.gibba029.stdout: "duration": 0.00061771299999999999 , 2021-03-13T22:40:33.564 INFO:teuthology.orchestra.run.gibba029.stdout: }, 2021-03-14T06:23:31.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:40:59.840 INFO:teuthology.orchestra.run.gibba007.stdout: , 2021-03-13T22:40:59.840 INFO:teuthology.orchestra.run.gibba007.stdout:{"epoch":262,"fsid":"e3b82a4b-4071-4f6d-bb66-37fca02e9e98","created":"2021-03-13T22:36:35.571730+0000","modified":"2021-03-13T22:40:59.379217+0000","last_up_change":"2021-03-13T22:40:06.667831+0000","last_in_change":"2021-03-13T22:40:23.365788+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-13T22:36:45.359621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-13T22:36:47.839429+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":18,"pg_placement_num":18,"pg_placement_num_target":18,"pg_num_target":18,"pg_num_pending":18,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"238","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"49","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2021-03-13T22:37:00.483627+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":6,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"3.16","ready_epoch":262,"last_epoch_started":261,"last_epoch_clean":261,"source_version":"232'244","target_version":"252'136"},"last_change":"262","last_force_op_resend":"0","last_force_op_resend_prenautilus":"262","last_force_op_resend_preluminous":"260","auid":0,"snap_mode":"selfmanaged","snap_seq":196,"snap_epoch":232,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"jerasure21profile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":true,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"7a3e0b1e-7522-4b86-baa1-538f5b5cad96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6800","nonce":29943},{"type":"v1","addr":"172.21.2.107:6801","nonce":29943}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6802","nonce":29943},{"type":"v1","addr":"172.21.2.107:6803","nonce":29943}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6806","nonce":29943},{"type":"v1","addr":"172.21.2.107:6807","nonce":29943}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6804","nonce":29943},{"type":"v1","addr":"172.21.2.107:6805","nonce":29943}]},"public_addr":"172.21.2.107:6801/29943","cluster_addr":"172.21.2.107:6803/29943","heartbeat_back_addr":"172.21.2.107:6807/29943","heartbeat_front_addr":"172.21.2.107:6805/29943","state":["exists","up"]},{"osd":1,"uuid":"f3916ee8-80d5-41b7-a878-36e23685a7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":221,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6808","nonce":29944},{"type":"v1","addr":"172.21.2.107:6809","nonce":29944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6810","nonce":29944},{"type":"v1","addr":"172.21.2.107:6811","nonce":29944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6814","nonce":29944},{"type":"v1","addr":"172.21.2.107:6817","nonce":29944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6812","nonce":29944},{"type":"v1","addr":"172.21.2.107:6813","nonce":29944}]},"public_addr":"172.21.2.107:6809/29944","cluster_addr":"172.21.2.107:6811/29944","heartbeat_back_addr":"172.21.2.107:6817/29944","heartbeat_front_addr":"172.21.2.107:6813/29944","state":["exists","up"]},{"osd":2,"uuid":"4fab38e2-7526-4918-aa8e-b6c86e6dfe3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6815","nonce":29945},{"type":"v1","addr":"172.21.2.107:6818","nonce":29945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6820","nonce":29945},{"type":"v1","addr":"172.21.2.107:6822","nonce":29945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6828","nonce":29945},{"type":"v1","addr":"172.21.2.107:6830","nonce":29945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6824","nonce":29945},{"type":"v1","addr":"172.21.2.107:6826","nonce":29945}]},"public_addr":"172.21.2.107:6818/29945","cluster_addr":"172.21.2.107:6822/29945","heartbeat_back_addr":"172.21.2.107:6830/29945","heartbeat_front_addr":"172.21.2.107:6826/29945","state":["exists","up"]},{"osd":3,"uuid":"5735c984-7507-400e-9189-24c912f0bcbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6816","nonce":29946},{"type":"v1","addr":"172.21.2.107:6819","nonce":29946}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6821","nonce":29946},{"type":"v1","addr":"172.21.2.107:6823","nonce":29946}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6829","nonce":29946},{"type":"v1","addr":"172.21.2.107:6831","nonce":29946}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.107:6825","nonce":29946},{"type":"v1","addr":"172.21.2.107:6827","nonce":29946}]},"public_addr":"172.21.2.107:6819/29946","cluster_addr":"172.21.2.107:6823/29946","heartbeat_back_addr":"172.21.2.107:6831/29946","heartbeat_front_addr":"172.21.2.107:6827/29946","state":["exists","up"]},{"osd":4,"uuid":"5a7b8a29-c6f6-46e7-b037-3764ffe2281a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6802","nonce":29896},{"type":"v1","addr":"172.21.2.129:6803","nonce":29896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6804","nonce":29896},{"type":"v1","addr":"172.21.2.129:6805","nonce":29896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6808","nonce":29896},{"type":"v1","addr":"172.21.2.129:6809","nonce":29896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6806","nonce":29896},{"type":"v1","addr":"172.21.2.129:6807","nonce":29896}]},"public_addr":"172.21.2.129:6803/29896","cluster_addr":"172.21.2.129:6805/29896","heartbeat_back_addr":"172.21.2.129:6809/29896","heartbeat_front_addr":"172.21.2.129:6807/29896","state":["exists","up"]},{"osd":5,"uuid":"32399b12-8f59-4e1b-97db-0ec10af6fdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":100,"up_from":210,"up_thru":259,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6810","nonce":44515},{"type":"v1","addr":"172.21.2.129:6811","nonce":44515}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6812","nonce":44515},{"type":"v1","addr":"172.21.2.129:6813","nonce":44515}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6816","nonce":44515},{"type":"v1","addr":"172.21.2.129:6817","nonce":44515}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6814","nonce":44515},{"type":"v1","addr":"172.21.2.129:6815","nonce":44515}]},"public_addr":"172.21.2.129:6811/44515","cluster_addr":"172.21.2.129:6813/44515","heartbeat_back_addr":"172.21.2.129:6817/44515","heartbeat_front_addr":"172.21.2.129:6815/44515","state":["exists","up"]},{"osd":6,"uuid":"b42429ba-39dd-4a1d-8e31-9b5ca5a6e7e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6818","nonce":29898},{"type":"v1","addr":"172.21.2.129:6819","nonce":29898}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6820","nonce":29898},{"type":"v1","addr":"172.21.2.129:6821","nonce":29898}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6824","nonce":29898},{"type":"v1","addr":"172.21.2.129:6825","nonce":29898}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6822","nonce":29898},{"type":"v1","addr":"172.21.2.129:6823","nonce":29898}]},"public_addr":"172.21.2.129:6819/29898","cluster_addr":"172.21.2.129:6821/29898","heartbeat_back_addr":"172.21.2.129:6825/29898","heartbeat_front_addr":"172.21.2.129:6823/29898","state":["exists","up"]},{"osd":7,"uuid":"d3f04855-f464-4a87-a3e5-5cf8ec791d41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":257,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6826","nonce":29899},{"type":"v1","addr":"172.21.2.129:6827","nonce":29899}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6828","nonce":29899},{"type":"v1","addr":"172.21.2.129:6829","nonce":29899}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6832","nonce":29899},{"type":"v1","addr":"172.21.2.129:6833","nonce":29899}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.129:6830","nonce":29899},{"type":"v1","addr":"172.21.2.129:6831","nonce":29899}]},"public_addr":"172.21.2.129:6827/29899","cluster_addr":"172.21.2.129:6829/29899","heartbeat_back_addr":"172.21.2.129:6833/29899","heartbeat_front_addr":"172.21.2.129:6831/29899","state":["exists","up"]},{"osd":8,"uuid":"cb0909a0-685d-4750-b92f-9fa17eb0843f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":258,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6800","nonce":30183},{"type":"v1","addr":"172.21.2.131:6801","nonce":30183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6802","nonce":30183},{"type":"v1","addr":"172.21.2.131:6803","nonce":30183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6806","nonce":30183},{"type":"v1","addr":"172.21.2.131:6807","nonce":30183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6804","nonce":30183},{"type":"v1","addr":"172.21.2.131:6805","nonce":30183}]},"public_addr":"172.21.2.131:6801/30183","cluster_addr":"172.21.2.131:6803/30183","heartbeat_back_addr":"172.21.2.131:6807/30183","heartbeat_front_addr":"172.21.2.131:6805/30183","state":["exists","up"]},{"osd":9,"uuid":"d536e4c4-f6dd-41a6-9fd1-b77818d1c2d6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":17,"last_clean_end":82,"up_from":137,"up_thru":252,"down_at":83,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6808","nonce":45775},{"type":"v1","addr":"172.21.2.131:6809","nonce":45775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6810","nonce":45775},{"type":"v1","addr":"172.21.2.131:6811","nonce":45775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6814","nonce":45775},{"type":"v1","addr":"172.21.2.131:6815","nonce":45775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6812","nonce":45775},{"type":"v1","addr":"172.21.2.131:6813","nonce":45775}]},"public_addr":"172.21.2.131:6809/45775","cluster_addr":"172.21.2.131:6811/45775","heartbeat_back_addr":"172.21.2.131:6815/45775","heartbeat_front_addr":"172.21.2.131:6813/45775","state":["exists","up"]},{"osd":10,"uuid":"0969fbad-59db-4d3b-a70c-fe76de2cc16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":259,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6816","nonce":30185},{"type":"v1","addr":"172.21.2.131:6817","nonce":30185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6818","nonce":30185},{"type":"v1","addr":"172.21.2.131:6819","nonce":30185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6825","nonce":30185},{"type":"v1","addr":"172.21.2.131:6827","nonce":30185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6821","nonce":30185},{"type":"v1","addr":"172.21.2.131:6823","nonce":30185}]},"public_addr":"172.21.2.131:6817/30185","cluster_addr":"172.21.2.131:6819/30185","heartbeat_back_addr":"172.21.2.131:6827/30185","heartbeat_front_addr":"172.21.2.131:6823/30185","state":["exists","up"]},{"osd":11,"uuid":"4a710f5f-bfba-45f4-a883-55c2559e4e5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":260,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6820","nonce":30186},{"type":"v1","addr":"172.21.2.131:6822","nonce":30186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6824","nonce":30186},{"type":"v1","addr":"172.21.2.131:6826","nonce":30186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6830","nonce":30186},{"type":"v1","addr":"172.21.2.131:6831","nonce":30186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.2.131:6828","nonce":30186},{"type":"v1","addr":"172.21.2.131:6829","nonce":30186}]},"public_addr":"172.21.2.131:6822/30186","cluster_addr":"172.21.2.131:6826/30186","heartbeat_back_addr":"172.21.2.131:6831/30186","heartbeat_front_addr":"172.21.2.131:6829/30186","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:44.919913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:45.897344+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:46.491396+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:45.014134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:45.388868+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:05.755361+0000","dead_epoch":101},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:46.363424+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:44.993774+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:46.397680+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:53.155126+0000","dead_epoch":83},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:45.945772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-13T22:40:46.096775+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.f","osds":[4,8,9,3,7,10]}],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":11,"to":5}]}],"pg_temp":[{"pgid":"3.9","osds":[8,10,7,9,5,1]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"},"jerasure21profile":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"4","m":"2","name":"jerasure21profile","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} , 2021-03-13T22:40:59.850 DEBUG:teuthology.orchestra.run.gibba007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-03-14T06:23:31.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:41:48.386 DEBUG:teuthology.orchestra.run.gibba007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2021-03-13T22:41:48.605 INFO:teuthology.orchestra.run.gibba007.stdout: , dumped all 2021-03-14T06:23:31.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:41:54.269 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2021-03-13T22:41:03.007691+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=41, tm_sec=3, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=41, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T22:41:54.270 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2021-03-13T22:41:13.847906+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=41, tm_sec=13, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=41, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T22:41:54.286 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2021-03-13T22:40:40.523974+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=40, tm_sec=40, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=22, tm_min=41, tm_sec=48, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:31.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T22:42:15.573 INFO:teuthology.misc:Shutting down mgr daemons... , 2021-03-13T22:42:15.574 DEBUG:tasks.ceph.mgr.x:waiting for process to exit , 2021-03-13T22:42:15.574 INFO:teuthology.orchestra.run:waiting for 300 2021-03-14T06:23:31.438 WARNING:scrape:Ignoring malparsed backtrace: next(self.gen) , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 277, in task , running.get() 2021-03-14T06:23:31.439 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks , suppress = manager.__exit__(*exc_info) , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ 2021-03-14T06:23:32.604 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x7fb5294885dd] , 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb5294887b8] 2021-03-14T06:23:32.605 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:56.627 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "sub_op_commit_rec", , 2021-03-13T14:48:56.627 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:42:46.387731+0000", , 2021-03-13T14:48:56.627 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.00016725799999999999 2021-03-14T06:23:32.606 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:56.684 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:56.684 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "throttled", , 2021-03-13T14:48:56.684 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:42:45.810185+0000", 2021-03-14T06:23:32.607 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:56.781 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:56.781 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "op_commit", , 2021-03-13T14:48:56.781 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:43:21.831256+0000", 2021-03-14T06:23:32.607 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:56.860 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:56.860 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "done", , 2021-03-13T14:48:56.860 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:43:22.076152+0000", 2021-03-14T06:23:32.608 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:56.970 INFO:teuthology.orchestra.run.gibba042.stdout: }, , 2021-03-13T14:48:56.971 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:56.971 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "reached_pg", 2021-03-14T06:23:32.608 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.042 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 4.7750000000000002e-05 , 2021-03-13T14:48:57.042 INFO:teuthology.orchestra.run.gibba042.stdout: }, , 2021-03-13T14:48:57.043 INFO:teuthology.orchestra.run.gibba042.stdout: { 2021-03-14T06:23:32.609 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.131 INFO:teuthology.orchestra.run.gibba042.stdout: "type_data": { , 2021-03-13T14:48:57.131 INFO:teuthology.orchestra.run.gibba042.stdout: "flag_point": "commit sent; apply or cleanup", , 2021-03-13T14:48:57.131 INFO:teuthology.orchestra.run.gibba042.stdout: "events": [ 2021-03-14T06:23:32.610 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.171 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:46:26.274560+0000", , 2021-03-13T14:48:57.171 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 8.3033000000000003e-05 , 2021-03-13T14:48:57.171 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:32.610 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.197 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "queued_for_pg", , 2021-03-13T14:48:57.197 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:47:37.330852+0000", , 2021-03-13T14:48:57.197 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 6.0006000000000001e-05 2021-03-14T06:23:32.611 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.221 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:57.221 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "all_read", , 2021-03-13T14:48:57.222 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:48:21.243952+0000", 2021-03-14T06:23:32.612 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.244 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:57.244 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "op_commit", , 2021-03-13T14:48:57.245 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T14:48:40.098422+0000", 2021-03-14T06:23:32.612 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.295 INFO:teuthology.orchestra.run.gibba042.stdout: "events": [ , 2021-03-13T14:48:57.295 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:57.295 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "initiated", 2021-03-14T06:23:32.613 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:57.321 INFO:teuthology.orchestra.run.gibba042.stdout: }, , 2021-03-13T14:48:57.322 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T14:48:57.322 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "done", 2021-03-14T06:23:32.614 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:48:59.621 INFO:tasks.thrashosds.thrasher:inning osd , 2021-03-13T14:48:59.621 INFO:tasks.thrashosds.thrasher:Adding osd 6 , 2021-03-13T14:48:59.621 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2021-03-14T06:23:32.618 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:49:21.269 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542234 got 60129542234 for osd.5 , 2021-03-13T14:49:21.270 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 , 2021-03-13T14:49:21.486 INFO:teuthology.orchestra.run.gibba040.stdout:60129542235 2021-03-14T06:23:32.620 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:49:51.341 INFO:teuthology.orchestra.run.gibba040.stdout:{ , 2021-03-13T14:49:51.341 INFO:teuthology.orchestra.run.gibba040.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " , 2021-03-13T14:49:51.342 INFO:teuthology.orchestra.run.gibba040.stdout:} 2021-03-14T06:23:32.621 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:50:12.201 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2021-03-13T14:48:57.464975+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=48, tm_sec=57, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=49, tm_sec=47, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T14:50:12.216 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2021-03-13T14:49:40.393965+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=49, tm_sec=40, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=49, tm_sec=47, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T14:50:12.216 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2021-03-13T14:49:28.045044+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=49, tm_sec=28, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=14, tm_min=49, tm_sec=47, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:32.622 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:50:59.709 INFO:teuthology.orchestra.run:waiting for 300 , 2021-03-13T14:50:59.764 INFO:tasks.ceph.mon.c:Stopped , 2021-03-13T14:50:59.764 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-03-14T06:23:32.623 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:51:33.361 DEBUG:teuthology.orchestra.run.gibba040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true , 2021-03-13T14:51:33.376 INFO:teuthology.orchestra.run.gibba040.stdout: remote refid st t when poll reach delay offset jitter , 2021-03-13T14:51:33.377 INFO:teuthology.orchestra.run.gibba040.stdout:============================================================================== 2021-03-14T06:23:32.623 WARNING:scrape:Ignoring malparsed backtrace: raise value.with_traceback(tb) , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 265, in thread 2021-03-14T06:23:32.994 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f110bb87308] , 2: /usr/lib64/ceph/libceph-common.so.2(+0x276522) [0x7f110bb87522] 2021-03-14T06:23:32.995 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.459 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.970170+0000", , 2021-03-13T16:53:15.459 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.00029248700000000002 , 2021-03-13T16:53:15.459 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:32.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.487 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.975719+0000", , 2021-03-13T16:53:15.487 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 6.0885999999999999e-05 , 2021-03-13T16:53:15.488 INFO:teuthology.orchestra.run.gibba042.stdout: }, 2021-03-14T06:23:32.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.508 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.975914+0000", , 2021-03-13T16:53:15.509 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.79420922400000005 , 2021-03-13T16:53:15.509 INFO:teuthology.orchestra.run.gibba042.stdout: } 2021-03-14T06:23:32.997 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.537 INFO:teuthology.orchestra.run.gibba042.stdout: }, , 2021-03-13T16:53:15.537 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T16:53:15.537 INFO:teuthology.orchestra.run.gibba042.stdout: "description": "osd_op(client.4572.0:3699 3.cs0 3:330cb8a4:::gibba04041646-819:head [setxattr _header (56) in=7b,truncate 3748126] snapc 0=[] ondisk+write+known_if_redirected e31)", 2021-03-14T06:23:32.997 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.561 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.79566825399999996, , 2021-03-13T16:53:15.561 INFO:teuthology.orchestra.run.gibba042.stdout: "type_data": { , 2021-03-13T16:53:15.562 INFO:teuthology.orchestra.run.gibba042.stdout: "flag_point": "started", 2021-03-14T06:23:33.000 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.583 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "initiated", , 2021-03-13T16:53:15.583 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.275735+0000", , 2021-03-13T16:53:15.583 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0 2021-03-14T06:23:33.001 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.604 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "all_read", , 2021-03-13T16:53:15.605 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.281151+0000", , 2021-03-13T16:53:15.605 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 6.72e-06 2021-03-14T06:23:33.002 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.629 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "dispatched", , 2021-03-13T16:53:15.629 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.284908+0000", , 2021-03-13T16:53:15.630 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 3.1690000000000001e-06 2021-03-14T06:23:33.003 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.677 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "queued_for_pg", , 2021-03-13T16:53:15.678 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.284965+0000", , 2021-03-13T16:53:15.678 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 2.0998000000000001e-05 2021-03-14T06:23:33.003 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.722 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "reached_pg", , 2021-03-13T16:53:15.722 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.285027+0000", , 2021-03-13T16:53:15.723 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 8.0200000000000001e-07 2021-03-14T06:23:33.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.754 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "reached_pg", , 2021-03-13T16:53:15.755 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.977569+0000", , 2021-03-13T16:53:15.755 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 5.7864e-05 2021-03-14T06:23:33.005 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.792 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "started", , 2021-03-13T16:53:15.793 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.980689+0000", , 2021-03-13T16:53:15.793 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.00059778299999999995 2021-03-14T06:23:33.005 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.814 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "sub_op_started", , 2021-03-13T16:53:15.814 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.982099+0000", , 2021-03-13T16:53:15.814 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.00086447599999999998 2021-03-14T06:23:33.006 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.838 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "started", , 2021-03-13T16:53:15.838 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:04.984886+0000", , 2021-03-13T16:53:15.838 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 0.000720679 2021-03-14T06:23:33.006 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.860 INFO:teuthology.orchestra.run.gibba042.stdout: "age": 108.568994202, , 2021-03-13T16:53:15.861 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 6.5373241609999999, , 2021-03-13T16:53:15.861 INFO:teuthology.orchestra.run.gibba042.stdout: "type_data": { 2021-03-14T06:23:33.007 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.903 INFO:teuthology.orchestra.run.gibba042.stdout: "age": 108.568947292, , 2021-03-13T16:53:15.904 INFO:teuthology.orchestra.run.gibba042.stdout: "duration": 6.5379626809999998, , 2021-03-13T16:53:15.904 INFO:teuthology.orchestra.run.gibba042.stdout: "type_data": { 2021-03-14T06:23:33.008 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.925 INFO:teuthology.orchestra.run.gibba042.stdout: "client": "client.4572", , 2021-03-13T16:53:15.925 INFO:teuthology.orchestra.run.gibba042.stdout: "client_addr": "172.21.2.140:0/314201197", , 2021-03-13T16:53:15.925 INFO:teuthology.orchestra.run.gibba042.stdout: "tid": 5046 2021-03-14T06:23:33.009 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:15.946 INFO:teuthology.orchestra.run.gibba042.stdout: { , 2021-03-13T16:53:15.946 INFO:teuthology.orchestra.run.gibba042.stdout: "event": "dispatched", , 2021-03-13T16:53:15.947 INFO:teuthology.orchestra.run.gibba042.stdout: "time": "2021-03-13T16:51:31.850972+0000", 2021-03-14T06:23:33.014 WARNING:scrape:Ignoring malparsed backtrace: Traceback (most recent call last): , File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run , File "/home/teuthworker/src/github.com_ceph_ceph-c_95bec98739256c1e2cefa7b63b1eff66d400d46c/qa/tasks/rados.py", line 265, in thread 2021-03-14T06:23:33.015 WARNING:scrape:Ignoring malparsed backtrace: dumped all , 2021-03-13T16:53:27.702 INFO:teuthology.orchestra.run.gibba040.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2021-03-13T16:53:27.046633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3033845653,"num_objects":1001,"num_object_clones":28,"num_object_copies":3003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":5,"num_read":2209,"num_read_kb":729649,"num_write":4777,"num_write_kb":1382252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":5910973512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9536,"ondisk_log_size":9536,"up":116,"acting":116,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":4696084,"kb_used_data":4505572,"kb_used_omap":17,"kb_used_meta":190446,"kb_avail":561534956,"statfs":{"total":579820584960,"available":575011794944,"internally_reserved":0,"allocated":4613705728,"data_stored":4663080966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17750,"internal_metadata":195017386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19206433,"num_objects":-11,"num_object_clones":-9,"num_object_copies":-33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-15,"num_objects_unfound":0,"num_objects_dirty":-11,"num_whiteouts":0,"num_read":-75,"num_read_kb":-5305,"num_write":-196,"num_write_kb":-54436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-66,"num_bytes_recovered":-225757812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001646"},"pg_stats":[{"pgid":"1.4","version":"0'0","reported_seq":"176","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874341+0000","last_change":"2021-03-13T16:52:38.952568+0000","last_active":"2021-03-13T16:53:22.874341+0000","last_peered":"2021-03-13T16:53:22.874341+0000","last_clean":"2021-03-13T16:53:22.874341+0000","last_became_active":"2021-03-13T16:52:38.952389+0000","last_became_peered":"2021-03-13T16:52:38.952389+0000","last_unstale":"2021-03-13T16:53:22.874341+0000","last_undegraded":"2021-03-13T16:53:22.874341+0000","last_fullsized":"2021-03-13T16:53:22.874341+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"155'327","reported_seq":"906","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874837+0000","last_change":"2021-03-13T16:53:14.877403+0000","last_active":"2021-03-13T16:53:22.874837+0000","last_peered":"2021-03-13T16:53:22.874837+0000","last_clean":"2021-03-13T16:53:22.874837+0000","last_became_active":"2021-03-13T16:52:12.868649+0000","last_became_peered":"2021-03-13T16:52:12.868649+0000","last_unstale":"2021-03-13T16:53:22.874837+0000","last_undegraded":"2021-03-13T16:53:22.874837+0000","last_fullsized":"2021-03-13T16:53:22.874837+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'287","last_scrub_stamp":"2021-03-13T16:52:32.784564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:32.784564+0000","log_size":327,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103390718,"num_objects":35,"num_object_clones":2,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":15,"num_write_kb":6641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.7","version":"0'0","reported_seq":"73","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872536+0000","last_change":"2021-03-13T16:53:22.579042+0000","last_active":"2021-03-13T16:53:22.872536+0000","last_peered":"2021-03-13T16:53:22.872536+0000","last_clean":"2021-03-13T16:53:22.872536+0000","last_became_active":"2021-03-13T16:53:06.881967+0000","last_became_peered":"2021-03-13T16:53:06.881967+0000","last_unstale":"2021-03-13T16:53:22.872536+0000","last_undegraded":"2021-03-13T16:53:22.872536+0000","last_fullsized":"2021-03-13T16:53:22.872536+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:53:22.578982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:53:22.578982+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"68","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.308630+0000","last_change":"2021-03-13T16:53:07.877168+0000","last_active":"2021-03-13T16:53:23.308630+0000","last_peered":"2021-03-13T16:53:23.308630+0000","last_clean":"2021-03-13T16:53:23.308630+0000","last_became_active":"2021-03-13T16:53:07.876909+0000","last_became_peered":"2021-03-13T16:53:07.876909+0000","last_unstale":"2021-03-13T16:53:23.308630+0000","last_undegraded":"2021-03-13T16:53:23.308630+0000","last_fullsized":"2021-03-13T16:53:23.308630+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:48.930840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:52:48.930840+0000","last_clean_scrub_stamp":"2021-03-13T16:52:48.930840+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"28","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.150211+0000","last_change":"2021-03-13T16:53:23.150159+0000","last_active":"2021-03-13T16:53:23.150211+0000","last_peered":"2021-03-13T16:53:23.150211+0000","last_clean":"2021-03-13T16:53:23.150211+0000","last_became_active":"2021-03-13T16:53:08.879157+0000","last_became_peered":"2021-03-13T16:53:08.879157+0000","last_unstale":"2021-03-13T16:53:23.150211+0000","last_undegraded":"2021-03-13T16:53:23.150211+0000","last_fullsized":"2021-03-13T16:53:23.150211+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:53:23.150079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:53:23.150079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"156'379","reported_seq":"229","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874864+0000","last_change":"2021-03-13T16:53:14.880018+0000","last_active":"2021-03-13T16:53:22.874864+0000","last_peered":"2021-03-13T16:53:22.874864+0000","last_clean":"2021-03-13T16:53:22.874864+0000","last_became_active":"2021-03-13T16:52:58.868387+0000","last_became_peered":"2021-03-13T16:52:58.868387+0000","last_unstale":"2021-03-13T16:53:22.874864+0000","last_undegraded":"2021-03-13T16:53:22.874864+0000","last_fullsized":"2021-03-13T16:53:22.874864+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'371","last_scrub_stamp":"2021-03-13T16:52:45.140618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:45.140618+0000","log_size":379,"ondisk_log_size":379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100464716,"num_objects":33,"num_object_clones":1,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":99,"num_read_kb":35398,"num_write":221,"num_write_kb":56202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":81981440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"1.6","version":"0'0","reported_seq":"24","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872474+0000","last_change":"2021-03-13T16:53:07.880596+0000","last_active":"2021-03-13T16:53:22.872474+0000","last_peered":"2021-03-13T16:53:22.872474+0000","last_clean":"2021-03-13T16:53:22.872474+0000","last_became_active":"2021-03-13T16:53:07.880340+0000","last_became_peered":"2021-03-13T16:53:07.880340+0000","last_unstale":"2021-03-13T16:53:22.872474+0000","last_undegraded":"2021-03-13T16:53:22.872474+0000","last_fullsized":"2021-03-13T16:53:22.872474+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"151'324","reported_seq":"1038","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872353+0000","last_change":"2021-03-13T16:53:14.880369+0000","last_active":"2021-03-13T16:53:22.872353+0000","last_peered":"2021-03-13T16:53:22.872353+0000","last_clean":"2021-03-13T16:53:22.872353+0000","last_became_active":"2021-03-13T16:53:09.881456+0000","last_became_peered":"2021-03-13T16:53:09.881456+0000","last_unstale":"2021-03-13T16:53:22.872353+0000","last_undegraded":"2021-03-13T16:53:22.872353+0000","last_fullsized":"2021-03-13T16:53:22.872353+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:50:49.900779+0000","log_size":324,"ondisk_log_size":324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":93526134,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":100,"num_read_kb":42696,"num_write":186,"num_write_kb":58439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":180035870,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.5","version":"0'0","reported_seq":"160","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874378+0000","last_change":"2021-03-13T16:52:12.383895+0000","last_active":"2021-03-13T16:53:22.874378+0000","last_peered":"2021-03-13T16:53:22.874378+0000","last_clean":"2021-03-13T16:53:22.874378+0000","last_became_active":"2021-03-13T16:50:37.865779+0000","last_became_peered":"2021-03-13T16:50:37.865779+0000","last_unstale":"2021-03-13T16:53:22.874378+0000","last_undegraded":"2021-03-13T16:53:22.874378+0000","last_fullsized":"2021-03-13T16:53:22.874378+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:12.383821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:52:12.383821+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"67","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872309+0000","last_change":"2021-03-13T16:53:07.879158+0000","last_active":"2021-03-13T16:53:22.872309+0000","last_peered":"2021-03-13T16:53:22.872309+0000","last_clean":"2021-03-13T16:53:22.872309+0000","last_became_active":"2021-03-13T16:53:07.878956+0000","last_became_peered":"2021-03-13T16:53:07.878956+0000","last_unstale":"2021-03-13T16:53:22.872309+0000","last_undegraded":"2021-03-13T16:53:22.872309+0000","last_fullsized":"2021-03-13T16:53:22.872309+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:44.991937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:44.991937+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"148'439","reported_seq":"1166","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874705+0000","last_change":"2021-03-13T16:53:14.877353+0000","last_active":"2021-03-13T16:53:22.874705+0000","last_peered":"2021-03-13T16:53:22.874705+0000","last_clean":"2021-03-13T16:53:22.874705+0000","last_became_active":"2021-03-13T16:52:12.865879+0000","last_became_peered":"2021-03-13T16:52:12.865879+0000","last_unstale":"2021-03-13T16:53:22.874705+0000","last_undegraded":"2021-03-13T16:53:22.874705+0000","last_fullsized":"2021-03-13T16:53:22.874705+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'412","last_scrub_stamp":"2021-03-13T16:52:55.874178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:55.874178+0000","log_size":439,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":109658737,"num_objects":36,"num_object_clones":1,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":12,"num_write_kb":4325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":"176","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874307+0000","last_change":"2021-03-13T16:53:08.880833+0000","last_active":"2021-03-13T16:53:22.874307+0000","last_peered":"2021-03-13T16:53:22.874307+0000","last_clean":"2021-03-13T16:53:22.874307+0000","last_became_active":"2021-03-13T16:53:08.880594+0000","last_became_peered":"2021-03-13T16:53:08.880594+0000","last_unstale":"2021-03-13T16:53:22.874307+0000","last_undegraded":"2021-03-13T16:53:22.874307+0000","last_fullsized":"2021-03-13T16:53:22.874307+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:51:47.410214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:51:47.410214+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"156'337","reported_seq":"1230","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874638+0000","last_change":"2021-03-13T16:53:14.878460+0000","last_active":"2021-03-13T16:53:22.874638+0000","last_peered":"2021-03-13T16:53:22.874638+0000","last_clean":"2021-03-13T16:53:22.874638+0000","last_became_active":"2021-03-13T16:53:10.865322+0000","last_became_peered":"2021-03-13T16:53:10.865322+0000","last_unstale":"2021-03-13T16:53:22.874638+0000","last_undegraded":"2021-03-13T16:53:22.874638+0000","last_fullsized":"2021-03-13T16:53:22.874638+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'290","last_scrub_stamp":"2021-03-13T16:52:09.370218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:09.370218+0000","log_size":337,"ondisk_log_size":337,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102792028,"num_objects":36,"num_object_clones":4,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":92,"num_read_kb":19204,"num_write":190,"num_write_kb":59161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":266314785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"1.0","version":"0'0","reported_seq":"174","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874267+0000","last_change":"2021-03-13T16:53:22.628332+0000","last_active":"2021-03-13T16:53:22.874267+0000","last_peered":"2021-03-13T16:53:22.874267+0000","last_clean":"2021-03-13T16:53:22.874267+0000","last_became_active":"2021-03-13T16:52:36.868001+0000","last_became_peered":"2021-03-13T16:52:36.868001+0000","last_unstale":"2021-03-13T16:53:22.874267+0000","last_undegraded":"2021-03-13T16:53:22.874267+0000","last_fullsized":"2021-03-13T16:53:22.874267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:53:22.628243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:53:22.628243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"165","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872472+0000","last_change":"2021-03-13T16:53:22.128154+0000","last_active":"2021-03-13T16:53:22.872472+0000","last_peered":"2021-03-13T16:53:22.872472+0000","last_clean":"2021-03-13T16:53:22.872472+0000","last_became_active":"2021-03-13T16:50:37.863974+0000","last_became_peered":"2021-03-13T16:50:37.863974+0000","last_unstale":"2021-03-13T16:53:22.872472+0000","last_undegraded":"2021-03-13T16:53:22.872472+0000","last_fullsized":"2021-03-13T16:53:22.872472+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:53:22.128107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:53:22.128107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"168","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874310+0000","last_change":"2021-03-13T16:52:36.872388+0000","last_active":"2021-03-13T16:53:22.874310+0000","last_peered":"2021-03-13T16:53:22.874310+0000","last_clean":"2021-03-13T16:53:22.874310+0000","last_became_active":"2021-03-13T16:52:36.872122+0000","last_became_peered":"2021-03-13T16:52:36.872122+0000","last_unstale":"2021-03-13T16:53:22.874310+0000","last_undegraded":"2021-03-13T16:53:22.874310+0000","last_fullsized":"2021-03-13T16:53:22.874310+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"151'379","reported_seq":"1123","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872304+0000","last_change":"2021-03-13T16:53:14.880311+0000","last_active":"2021-03-13T16:53:22.872304+0000","last_peered":"2021-03-13T16:53:22.872304+0000","last_clean":"2021-03-13T16:53:22.872304+0000","last_became_active":"2021-03-13T16:53:09.886616+0000","last_became_peered":"2021-03-13T16:53:09.886616+0000","last_unstale":"2021-03-13T16:53:22.872304+0000","last_undegraded":"2021-03-13T16:53:22.872304+0000","last_fullsized":"2021-03-13T16:53:22.872304+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'333","last_scrub_stamp":"2021-03-13T16:52:08.983945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:08.983945+0000","log_size":379,"ondisk_log_size":379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102953862,"num_objects":34,"num_object_clones":1,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":108,"num_read_kb":49554,"num_write":196,"num_write_kb":57188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":173786754,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":"173","reported_epoch":"164","state":"active+clean","last_fresh":"2021-03-13T16:53:24.063753+0000","last_change":"2021-03-13T16:53:24.063753+0000","last_active":"2021-03-13T16:53:24.063753+0000","last_peered":"2021-03-13T16:53:24.063753+0000","last_clean":"2021-03-13T16:53:24.063753+0000","last_became_active":"2021-03-13T16:51:22.849746+0000","last_became_peered":"2021-03-13T16:51:22.849746+0000","last_unstale":"2021-03-13T16:53:24.063753+0000","last_undegraded":"2021-03-13T16:53:24.063753+0000","last_fullsized":"2021-03-13T16:53:24.063753+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:53:24.063693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:53:24.063693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"164'313","reported_seq":"952","reported_epoch":"164","state":"active+clean","last_fresh":"2021-03-13T16:53:23.902439+0000","last_change":"2021-03-13T16:53:23.902650+0000","last_active":"2021-03-13T16:53:23.902439+0000","last_peered":"2021-03-13T16:53:23.902439+0000","last_clean":"2021-03-13T16:53:23.902439+0000","last_became_active":"2021-03-13T16:53:23.879657+0000","last_became_peered":"2021-03-13T16:53:23.879657+0000","last_unstale":"2021-03-13T16:53:23.902439+0000","last_undegraded":"2021-03-13T16:53:23.902439+0000","last_fullsized":"2021-03-13T16:53:23.902439+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'268","last_scrub_stamp":"2021-03-13T16:52:08.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:08.317836+0000","log_size":313,"ondisk_log_size":313,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":99104246,"num_objects":32,"num_object_clones":2,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":92,"num_read_kb":39980,"num_write":184,"num_write_kb":57476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":202707752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.2","version":"0'0","reported_seq":"93","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872304+0000","last_change":"2021-03-13T16:52:11.992723+0000","last_active":"2021-03-13T16:53:22.872304+0000","last_peered":"2021-03-13T16:53:22.872304+0000","last_clean":"2021-03-13T16:53:22.872304+0000","last_became_active":"2021-03-13T16:52:00.867538+0000","last_became_peered":"2021-03-13T16:52:00.867538+0000","last_unstale":"2021-03-13T16:53:22.872304+0000","last_undegraded":"2021-03-13T16:53:22.872304+0000","last_fullsized":"2021-03-13T16:53:22.872304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:11.992667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:52:11.992667+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"168","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874689+0000","last_change":"2021-03-13T16:52:38.955755+0000","last_active":"2021-03-13T16:53:22.874689+0000","last_peered":"2021-03-13T16:53:22.874689+0000","last_clean":"2021-03-13T16:53:22.874689+0000","last_became_active":"2021-03-13T16:52:38.955503+0000","last_became_peered":"2021-03-13T16:52:38.955503+0000","last_unstale":"2021-03-13T16:53:22.874689+0000","last_undegraded":"2021-03-13T16:53:22.874689+0000","last_fullsized":"2021-03-13T16:53:22.874689+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"24","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872504+0000","last_change":"2021-03-13T16:53:07.880308+0000","last_active":"2021-03-13T16:53:22.872504+0000","last_peered":"2021-03-13T16:53:22.872504+0000","last_clean":"2021-03-13T16:53:22.872504+0000","last_became_active":"2021-03-13T16:53:07.880123+0000","last_became_peered":"2021-03-13T16:53:07.880123+0000","last_unstale":"2021-03-13T16:53:22.872504+0000","last_undegraded":"2021-03-13T16:53:22.872504+0000","last_fullsized":"2021-03-13T16:53:22.872504+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"148'423","reported_seq":"1070","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872568+0000","last_change":"2021-03-13T16:53:14.877078+0000","last_active":"2021-03-13T16:53:22.872568+0000","last_peered":"2021-03-13T16:53:22.872568+0000","last_clean":"2021-03-13T16:53:22.872568+0000","last_became_active":"2021-03-13T16:52:12.866524+0000","last_became_peered":"2021-03-13T16:52:12.866524+0000","last_unstale":"2021-03-13T16:53:22.872568+0000","last_undegraded":"2021-03-13T16:53:22.872568+0000","last_fullsized":"2021-03-13T16:53:22.872568+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'375","last_scrub_stamp":"2021-03-13T16:52:35.393905+0000","last_deep_scrub":"116'375","last_deep_scrub_stamp":"2021-03-13T16:52:35.393905+0000","last_clean_scrub_stamp":"2021-03-13T16:52:35.393905+0000","log_size":423,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114037572,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":16,"num_read_kb":24894,"num_write":12,"num_write_kb":8166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":"162","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872345+0000","last_change":"2021-03-13T16:52:07.996458+0000","last_active":"2021-03-13T16:53:22.872345+0000","last_peered":"2021-03-13T16:53:22.872345+0000","last_clean":"2021-03-13T16:53:22.872345+0000","last_became_active":"2021-03-13T16:50:37.864106+0000","last_became_peered":"2021-03-13T16:50:37.864106+0000","last_unstale":"2021-03-13T16:53:22.872345+0000","last_undegraded":"2021-03-13T16:53:22.872345+0000","last_fullsized":"2021-03-13T16:53:22.872345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:07.996406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:36.852586+0000","last_clean_scrub_stamp":"2021-03-13T16:52:07.996406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"58","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.308502+0000","last_change":"2021-03-13T16:52:43.874960+0000","last_active":"2021-03-13T16:53:23.308502+0000","last_peered":"2021-03-13T16:53:23.308502+0000","last_clean":"2021-03-13T16:53:23.308502+0000","last_became_active":"2021-03-13T16:52:38.952802+0000","last_became_peered":"2021-03-13T16:52:38.952802+0000","last_unstale":"2021-03-13T16:53:23.308502+0000","last_undegraded":"2021-03-13T16:53:23.308502+0000","last_fullsized":"2021-03-13T16:53:23.308502+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:43.874913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:43.874913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"148'408","reported_seq":"1190","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.308595+0000","last_change":"2021-03-13T16:53:14.871250+0000","last_active":"2021-03-13T16:53:23.308595+0000","last_peered":"2021-03-13T16:53:23.308595+0000","last_clean":"2021-03-13T16:53:23.308595+0000","last_became_active":"2021-03-13T16:52:12.868203+0000","last_became_peered":"2021-03-13T16:52:12.868203+0000","last_unstale":"2021-03-13T16:53:23.308595+0000","last_undegraded":"2021-03-13T16:53:23.308595+0000","last_fullsized":"2021-03-13T16:53:23.308595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'362","last_scrub_stamp":"2021-03-13T16:52:31.814791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:31.814791+0000","log_size":408,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":102257517,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":16,"num_read_kb":16899,"num_write":16,"num_write_kb":3983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.9","version":"155'382","reported_seq":"964","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872273+0000","last_change":"2021-03-13T16:53:14.880022+0000","last_active":"2021-03-13T16:53:22.872273+0000","last_peered":"2021-03-13T16:53:22.872273+0000","last_clean":"2021-03-13T16:53:22.872273+0000","last_became_active":"2021-03-13T16:52:12.866401+0000","last_became_peered":"2021-03-13T16:52:12.866401+0000","last_unstale":"2021-03-13T16:53:22.872273+0000","last_undegraded":"2021-03-13T16:53:22.872273+0000","last_fullsized":"2021-03-13T16:53:22.872273+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'327","last_scrub_stamp":"2021-03-13T16:52:31.050668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:31.050668+0000","log_size":382,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":103366231,"num_objects":36,"num_object_clones":2,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":8,"num_read_kb":6,"num_write":31,"num_write_kb":8451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"1.8","version":"0'0","reported_seq":"58","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.308429+0000","last_change":"2021-03-13T16:52:44.885325+0000","last_active":"2021-03-13T16:53:23.308429+0000","last_peered":"2021-03-13T16:53:23.308429+0000","last_clean":"2021-03-13T16:53:23.308429+0000","last_became_active":"2021-03-13T16:52:38.953328+0000","last_became_peered":"2021-03-13T16:52:38.953328+0000","last_unstale":"2021-03-13T16:53:23.308429+0000","last_undegraded":"2021-03-13T16:53:23.308429+0000","last_fullsized":"2021-03-13T16:53:23.308429+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:44.885274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:44.885274+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"164'284","reported_seq":"1155","reported_epoch":"164","state":"active+clean","last_fresh":"2021-03-13T16:53:24.619324+0000","last_change":"2021-03-13T16:53:24.619324+0000","last_active":"2021-03-13T16:53:24.619324+0000","last_peered":"2021-03-13T16:53:24.619324+0000","last_clean":"2021-03-13T16:53:24.619324+0000","last_became_active":"2021-03-13T16:53:23.880050+0000","last_became_peered":"2021-03-13T16:53:23.880050+0000","last_unstale":"2021-03-13T16:53:24.619324+0000","last_undegraded":"2021-03-13T16:53:24.619324+0000","last_fullsized":"2021-03-13T16:53:24.619324+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'284","last_scrub_stamp":"2021-03-13T16:53:24.619225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:53:24.619225+0000","log_size":284,"ondisk_log_size":284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":153978730,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":124,"num_read_kb":27531,"num_write":347,"num_write_kb":90133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":204,"num_bytes_recovered":565920439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"1.9","version":"0'0","reported_seq":"184","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874266+0000","last_change":"2021-03-13T16:52:55.864915+0000","last_active":"2021-03-13T16:53:22.874266+0000","last_peered":"2021-03-13T16:53:22.874266+0000","last_clean":"2021-03-13T16:53:22.874266+0000","last_became_active":"2021-03-13T16:52:55.864479+0000","last_became_peered":"2021-03-13T16:52:55.864479+0000","last_unstale":"2021-03-13T16:53:22.874266+0000","last_undegraded":"2021-03-13T16:53:22.874266+0000","last_fullsized":"2021-03-13T16:53:22.874266+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:52:10.380327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:35.851482+0000","last_clean_scrub_stamp":"2021-03-13T16:52:10.380327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"154'338","reported_seq":"1141","reported_epoch":"163","state":"active+clean+wait","last_fresh":"2021-03-13T16:53:22.872395+0000","last_change":"2021-03-13T16:53:17.875376+0000","last_active":"2021-03-13T16:53:22.872395+0000","last_peered":"2021-03-13T16:53:22.872395+0000","last_clean":"2021-03-13T16:53:22.872395+0000","last_became_active":"2021-03-13T16:53:17.873173+0000","last_became_peered":"2021-03-13T16:53:17.873173+0000","last_unstale":"2021-03-13T16:53:22.872395+0000","last_undegraded":"2021-03-13T16:53:22.872395+0000","last_fullsized":"2021-03-13T16:53:22.872395+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'307","last_scrub_stamp":"2021-03-13T16:52:43.154854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:43.154854+0000","log_size":338,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":156901595,"num_objects":51,"num_object_clones":2,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":168,"num_read_kb":84095,"num_write":316,"num_write_kb":97090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":510182681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.c","version":"151'319","reported_seq":"296","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874754+0000","last_change":"2021-03-13T16:53:14.877567+0000","last_active":"2021-03-13T16:53:22.874754+0000","last_peered":"2021-03-13T16:53:22.874754+0000","last_clean":"2021-03-13T16:53:22.874754+0000","last_became_active":"2021-03-13T16:53:09.881625+0000","last_became_peered":"2021-03-13T16:53:09.881625+0000","last_unstale":"2021-03-13T16:53:22.874754+0000","last_undegraded":"2021-03-13T16:53:22.874754+0000","last_fullsized":"2021-03-13T16:53:22.874754+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'235","last_scrub_stamp":"2021-03-13T16:52:08.244152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:08.244152+0000","log_size":319,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":141679275,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":150,"num_read_kb":58134,"num_write":300,"num_write_kb":89956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":149,"num_bytes_recovered":402200649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.d","version":"156'412","reported_seq":"1009","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874462+0000","last_change":"2021-03-13T16:53:14.880890+0000","last_active":"2021-03-13T16:53:22.874462+0000","last_peered":"2021-03-13T16:53:22.874462+0000","last_clean":"2021-03-13T16:53:22.874462+0000","last_became_active":"2021-03-13T16:53:04.873806+0000","last_became_peered":"2021-03-13T16:53:04.873806+0000","last_unstale":"2021-03-13T16:53:22.874462+0000","last_undegraded":"2021-03-13T16:53:22.874462+0000","last_fullsized":"2021-03-13T16:53:22.874462+0000","mapping_epoch":145,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":146,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'330","last_scrub_stamp":"2021-03-13T16:52:45.351808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:45.351808+0000","log_size":412,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":189007785,"num_objects":64,"num_object_clones":3,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":164,"num_read_kb":18254,"num_write":401,"num_write_kb":119168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":207469399,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.e","version":"155'430","reported_seq":"487","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874413+0000","last_change":"2021-03-13T16:53:14.877914+0000","last_active":"2021-03-13T16:53:22.874413+0000","last_peered":"2021-03-13T16:53:22.874413+0000","last_clean":"2021-03-13T16:53:22.874413+0000","last_became_active":"2021-03-13T16:53:02.866368+0000","last_became_peered":"2021-03-13T16:53:02.866368+0000","last_unstale":"2021-03-13T16:53:22.874413+0000","last_undegraded":"2021-03-13T16:53:22.874413+0000","last_fullsized":"2021-03-13T16:53:22.874413+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"111'363","last_scrub_stamp":"2021-03-13T16:52:32.059760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:32.059760+0000","log_size":430,"ondisk_log_size":430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":205128670,"num_objects":66,"num_object_clones":1,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":184,"num_read_kb":52982,"num_write":437,"num_write_kb":117945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":241,"num_bytes_recovered":615133049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.f","version":"155'349","reported_seq":"943","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874564+0000","last_change":"2021-03-13T16:53:14.878844+0000","last_active":"2021-03-13T16:53:22.874564+0000","last_peered":"2021-03-13T16:53:22.874564+0000","last_clean":"2021-03-13T16:53:22.874564+0000","last_became_active":"2021-03-13T16:51:50.851295+0000","last_became_peered":"2021-03-13T16:51:50.851295+0000","last_unstale":"2021-03-13T16:53:22.874564+0000","last_undegraded":"2021-03-13T16:53:22.874564+0000","last_fullsized":"2021-03-13T16:53:22.874564+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'318","last_scrub_stamp":"2021-03-13T16:52:48.315778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:48.315778+0000","log_size":349,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":193687958,"num_objects":64,"num_object_clones":1,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":1,"num_read":175,"num_read_kb":63565,"num_write":331,"num_write_kb":101716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":202079671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.10","version":"151'366","reported_seq":"215","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872325+0000","last_change":"2021-03-13T16:53:14.873601+0000","last_active":"2021-03-13T16:53:22.872325+0000","last_peered":"2021-03-13T16:53:22.872325+0000","last_clean":"2021-03-13T16:53:22.872325+0000","last_became_active":"2021-03-13T16:53:09.884488+0000","last_became_peered":"2021-03-13T16:53:09.884488+0000","last_unstale":"2021-03-13T16:53:22.872325+0000","last_undegraded":"2021-03-13T16:53:22.872325+0000","last_fullsized":"2021-03-13T16:53:22.872325+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'333","last_scrub_stamp":"2021-03-13T16:52:08.983945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:08.983945+0000","log_size":366,"ondisk_log_size":366,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92914943,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":97,"num_read_kb":30065,"num_write":194,"num_write_kb":54205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":279705159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.11","version":"152'385","reported_seq":"376","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874454+0000","last_change":"2021-03-13T16:53:14.875267+0000","last_active":"2021-03-13T16:53:22.874454+0000","last_peered":"2021-03-13T16:53:22.874454+0000","last_clean":"2021-03-13T16:53:22.874454+0000","last_became_active":"2021-03-13T16:53:10.865877+0000","last_became_peered":"2021-03-13T16:53:10.865877+0000","last_unstale":"2021-03-13T16:53:22.874454+0000","last_undegraded":"2021-03-13T16:53:22.874454+0000","last_fullsized":"2021-03-13T16:53:22.874454+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:50:49.900779+0000","log_size":385,"ondisk_log_size":385,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114264525,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":1,"num_read":102,"num_read_kb":23400,"num_write":234,"num_write_kb":66034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":147,"num_bytes_recovered":415818197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.12","version":"156'391","reported_seq":"287","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874420+0000","last_change":"2021-03-13T16:53:14.877677+0000","last_active":"2021-03-13T16:53:22.874420+0000","last_peered":"2021-03-13T16:53:22.874420+0000","last_clean":"2021-03-13T16:53:22.874420+0000","last_became_active":"2021-03-13T16:52:28.870843+0000","last_became_peered":"2021-03-13T16:52:28.870843+0000","last_unstale":"2021-03-13T16:53:22.874420+0000","last_undegraded":"2021-03-13T16:53:22.874420+0000","last_fullsized":"2021-03-13T16:53:22.874420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'290","last_scrub_stamp":"2021-03-13T16:52:09.370218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:09.370218+0000","log_size":391,"ondisk_log_size":391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":94300566,"num_objects":31,"num_object_clones":1,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":93,"num_read_kb":14183,"num_write":220,"num_write_kb":59534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":379100254,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.13","version":"153'315","reported_seq":"316","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874375+0000","last_change":"2021-03-13T16:53:14.874953+0000","last_active":"2021-03-13T16:53:22.874375+0000","last_peered":"2021-03-13T16:53:22.874375+0000","last_clean":"2021-03-13T16:53:22.874375+0000","last_became_active":"2021-03-13T16:53:11.869154+0000","last_became_peered":"2021-03-13T16:53:11.869154+0000","last_unstale":"2021-03-13T16:53:22.874375+0000","last_undegraded":"2021-03-13T16:53:22.874375+0000","last_fullsized":"2021-03-13T16:53:22.874375+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'268","last_scrub_stamp":"2021-03-13T16:52:08.317836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:08.317836+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":87721469,"num_objects":28,"num_object_clones":1,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":86,"num_read_kb":32281,"num_write":187,"num_write_kb":49833,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":146307524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.14","version":"156'327","reported_seq":"413","reported_epoch":"164","state":"active+clean","last_fresh":"2021-03-13T16:53:25.032524+0000","last_change":"2021-03-13T16:53:25.032524+0000","last_active":"2021-03-13T16:53:25.032524+0000","last_peered":"2021-03-13T16:53:25.032524+0000","last_clean":"2021-03-13T16:53:25.032524+0000","last_became_active":"2021-03-13T16:53:09.881230+0000","last_became_peered":"2021-03-13T16:53:09.881230+0000","last_unstale":"2021-03-13T16:53:25.032524+0000","last_undegraded":"2021-03-13T16:53:25.032524+0000","last_fullsized":"2021-03-13T16:53:25.032524+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"156'327","last_scrub_stamp":"2021-03-13T16:53:25.032462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:53:25.032462+0000","log_size":327,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81648404,"num_objects":29,"num_object_clones":2,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.15","version":"151'379","reported_seq":"383","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874394+0000","last_change":"2021-03-13T16:53:14.874637+0000","last_active":"2021-03-13T16:53:22.874394+0000","last_peered":"2021-03-13T16:53:22.874394+0000","last_clean":"2021-03-13T16:53:22.874394+0000","last_became_active":"2021-03-13T16:53:09.883438+0000","last_became_peered":"2021-03-13T16:53:09.883438+0000","last_unstale":"2021-03-13T16:53:22.874394+0000","last_undegraded":"2021-03-13T16:53:22.874394+0000","last_fullsized":"2021-03-13T16:53:22.874394+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:50:49.900779+0000","log_size":379,"ondisk_log_size":379,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114195675,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":104,"num_read_kb":35013,"num_write":209,"num_write_kb":58954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":151,"num_bytes_recovered":406644297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.16","version":"148'292","reported_seq":"432","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872588+0000","last_change":"2021-03-13T16:53:14.877085+0000","last_active":"2021-03-13T16:53:22.872588+0000","last_peered":"2021-03-13T16:53:22.872588+0000","last_clean":"2021-03-13T16:53:22.872588+0000","last_became_active":"2021-03-13T16:53:11.868441+0000","last_became_peered":"2021-03-13T16:53:11.868441+0000","last_unstale":"2021-03-13T16:53:22.872588+0000","last_undegraded":"2021-03-13T16:53:22.872588+0000","last_fullsized":"2021-03-13T16:53:22.872588+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'283","last_scrub_stamp":"2021-03-13T16:52:45.107840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:45.107840+0000","log_size":292,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":74498495,"num_objects":24,"num_object_clones":1,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":9243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":74498495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.17","version":"148'440","reported_seq":"500","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.874872+0000","last_change":"2021-03-13T16:53:14.877973+0000","last_active":"2021-03-13T16:53:22.874872+0000","last_peered":"2021-03-13T16:53:22.874872+0000","last_clean":"2021-03-13T16:53:22.874872+0000","last_became_active":"2021-03-13T16:52:17.875546+0000","last_became_peered":"2021-03-13T16:52:17.875546+0000","last_unstale":"2021-03-13T16:53:22.874872+0000","last_undegraded":"2021-03-13T16:53:22.874872+0000","last_fullsized":"2021-03-13T16:53:22.874872+0000","mapping_epoch":98,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:50:49.900779+0000","log_size":440,"ondisk_log_size":440,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":123973677,"num_objects":38,"num_object_clones":2,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":114,"num_read_kb":39788,"num_write":277,"num_write_kb":74076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":144098295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.18","version":"158'450","reported_seq":"616","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:23.308480+0000","last_change":"2021-03-13T16:53:16.883989+0000","last_active":"2021-03-13T16:53:23.308480+0000","last_peered":"2021-03-13T16:53:23.308480+0000","last_clean":"2021-03-13T16:53:23.308480+0000","last_became_active":"2021-03-13T16:53:16.869990+0000","last_became_peered":"2021-03-13T16:53:16.869990+0000","last_unstale":"2021-03-13T16:53:23.308480+0000","last_undegraded":"2021-03-13T16:53:23.308480+0000","last_fullsized":"2021-03-13T16:53:23.308480+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:50:49.900779+0000","log_size":450,"ondisk_log_size":450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":101123843,"num_objects":35,"num_object_clones":1,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":103,"num_read_kb":21721,"num_write":249,"num_write_kb":71817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":248,"num_bytes_recovered":571552054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]},{"pgid":"3.19","version":"148'348","reported_seq":"1113","reported_epoch":"163","state":"active+clean","last_fresh":"2021-03-13T16:53:22.872269+0000","last_change":"2021-03-13T16:53:14.880083+0000","last_active":"2021-03-13T16:53:22.872269+0000","last_peered":"2021-03-13T16:53:22.872269+0000","last_clean":"2021-03-13T16:53:22.872269+0000","last_became_active":"2021-03-13T16:53:01.865674+0000","last_became_peered":"2021-03-13T16:53:01.865674+0000","last_unstale":"2021-03-13T16:53:22.872269+0000","last_undegraded":"2021-03-13T16:53:22.872269+0000","last_fullsized":"2021-03-13T16:53:22.872269+0000","mapping_epoch":142,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":143,"parent":"0.0","parent_split_bits":5,"last_scrub":"102'327","last_scrub_stamp":"2021-03-13T16:52:35.014193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-13T16:50:49.900779+0000","last_clean_scrub_stamp":"2021-03-13T16:52:35.014193+0000","log_size":348,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":77268282,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":9,"num_write_kb":2516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":85436748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"33","length":"1"},{"start":"37","length":"1"},{"start":"3a","length":"1"},{"start":"3c","length":"2"},{"start":"3f","length":"2"},{"start":"44","length":"2"},{"start":"48","length":"3"},{"start":"4c","length":"1"},{"start":"4e","length":"4"},{"start":"53","length":"1"},{"start":"59","length":"2"},{"start":"61","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":3033845653,"num_objects":1001,"num_object_clones":28,"num_object_copies":3003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1001,"num_whiteouts":5,"num_read":2209,"num_read_kb":729649,"num_write":4777,"num_write_kb":1382252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2215,"num_bytes_recovered":5910973512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4707753984,"data_stored":4770754560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9536,"ondisk_log_size":9536,"up":78,"acting":78,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":22,"acting":22,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1113216,"kb_used_data":1067832,"kb_used_omap":0,"kb_used_meta":45376,"kb_avail":93258624,"statfs":{"total":96636764160,"available":95496830976,"internally_reserved":0,"allocated":1093459968,"data_stored":1108091905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46465024},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.72999999999999998,"15min":0.73199999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":0.995},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.248,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.86699999999999999}]},{"osd":1,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.69499999999999995,"15min":0.68100000000000005},"min":{"1min":0.39700000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.74399999999999999,"15min":0.73899999999999999},"min":{"1min":0.46400000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.84299999999999997}]},{"osd":2,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.80200000000000005,"15min":0.79600000000000004},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.202},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.79100000000000004,"15min":0.78200000000000003},"min":{"1min":0.48699999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.95599999999999996}]},{"osd":3,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.79300000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.90100000000000002,"15min":0.90200000000000002},"min":{"1min":0.622,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.1359999999999999}]},{"osd":4,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.69599999999999995,"15min":0.69799999999999995},"min":{"1min":0.50900000000000001,"5min":0.436,"15min":0.436},"max":{"1min":0.92400000000000004,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.749,"15min":0.74399999999999999},"min":{"1min":0.56200000000000006,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.173,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":1.24}]},{"osd":5,"last update":"Sat Mar 13 16:52:36 2021","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.83299999999999996,"15min":0.83699999999999997},"min":{"1min":0.53500000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0429999999999999,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":1.391},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.63600000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.0620000000000001,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.0800000000000001}]},{"osd":6,"last update":"Sat Mar 13 16:51:35 2021","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":1.056},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2669999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"last":1.2749999999999999}]}]},{"osd":6,"up_from":147,"seq":631360192517,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":596596,"kb_used_data":586156,"kb_used_omap":17,"kb_used_meta":10414,"kb_avail":93775244,"statfs":{"total":96636764160,"available":96025849856,"internally_reserved":0,"allocated":600223744,"data_stored":605164545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17750,"internal_metadata":10664618},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":758824,"kb_used_data":723296,"kb_used_omap":0,"kb_used_meta":35520,"kb_avail":93613016,"statfs":{"total":96636764160,"available":95859728384,"internally_reserved":0,"allocated":740655104,"data_stored":745505793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36372480},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.501},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.53700000000000003}]},{"osd":2,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.60399999999999998,"15min":0.60799999999999998},"min":{"1min":0.47299999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53800000000000003,"15min":0.54000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70199999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56000000000000005}]},{"osd":3,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.622,"15min":0.625},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.0509999999999999,"5min":1.0509999999999999,"15min":1.0509999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.68600000000000005,"15min":0.68700000000000006},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.91400000000000003}]},{"osd":4,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.90200000000000002,"15min":0.91200000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.431,"5min":1.452,"15min":1.452},"last":1.4159999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.90200000000000002,"15min":0.91200000000000003},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.3740000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.98099999999999998,"15min":0.995},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.2749999999999999,"5min":1.341,"15min":1.341},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.95599999999999996,"15min":0.96999999999999997},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.3240000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":1.1930000000000001}]},{"osd":6,"last update":"Sat Mar 13 16:51:37 2021","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.407,"5min":1.407,"15min":1.407},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.048,"15min":1.048},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.4370000000000001}]},{"osd":7,"last update":"Sat Mar 13 16:52:39 2021","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.026,"15min":1.042},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.4910000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.085,"15min":1.097},"min":{"1min":0.68700000000000006,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.3420000000000001,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.208}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":839052,"kb_used_data":802948,"kb_used_omap":0,"kb_used_meta":36096,"kb_avail":93532788,"statfs":{"total":96636764160,"available":95777574912,"internally_reserved":0,"allocated":822218752,"data_stored":830800897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36962304},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69599999999999995,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54400000000000004,"15min":0.54200000000000004},"min":{"1min":0.39100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76900000000000002,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.63300000000000001}]},{"osd":2,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38200000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.379,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82999999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.78300000000000003}]},{"osd":3,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63200000000000001,"15min":0.63400000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.82099999999999995,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.65200000000000002,"15min":0.66000000000000003},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.79900000000000004,"5min":1.135,"15min":1.135},"last":0.58499999999999996}]},{"osd":4,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.91400000000000003,"15min":0.90800000000000003},"min":{"1min":0.52900000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.4359999999999999,"5min":1.806,"15min":1.806},"last":1.175},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92300000000000004,"15min":0.92400000000000004},"min":{"1min":0.60299999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.3580000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.1459999999999999}]},{"osd":5,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96899999999999997,"15min":0.97599999999999998},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.571,"5min":1.925,"15min":1.925},"last":1.242},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.98399999999999999,"15min":0.98999999999999999},"min":{"1min":0.66800000000000004,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.381,"5min":2.02,"15min":2.02},"last":1.359}]},{"osd":6,"last update":"Sat Mar 13 16:51:36 2021","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":1.252}]},{"osd":7,"last update":"Sat Mar 13 16:52:38 2021","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.054,"15min":1.0580000000000001},"min":{"1min":0.72699999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.448,"5min":1.76,"15min":1.76},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.016,"5min":1.0429999999999999,"15min":1.0469999999999999},"min":{"1min":0.72799999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4370000000000001,"5min":1.907,"15min":1.907},"last":1.3100000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":693500,"kb_used_data":664436,"kb_used_omap":0,"kb_used_meta":29056,"kb_avail":93678340,"statfs":{"total":96636764160,"available":95926620160,"internally_reserved":0,"allocated":680382464,"data_stored":687391745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29753344},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.61699999999999999,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.51700000000000002,"15min":0.52300000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68500000000000005,"5min":0.873,"15min":0.873},"last":0.54100000000000004}]},{"osd":1,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.40699999999999997,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.78600000000000003,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67300000000000004,"15min":0.67600000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.90000000000000002,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.78900000000000003}]},{"osd":3,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89600000000000002,"5min":1.198,"15min":1.198},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70799999999999996,"15min":0.71599999999999997},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0089999999999999,"5min":1.107,"15min":1.107},"last":0.73499999999999999}]},{"osd":4,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.95199999999999996,"15min":0.95899999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.492,"5min":1.532,"15min":1.532},"last":1.351},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.93100000000000005,"15min":0.92500000000000004},"min":{"1min":0.58599999999999997,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.466,"5min":1.466,"15min":1.466},"last":1.4259999999999999}]},{"osd":5,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.58999999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.458,"5min":1.506,"15min":1.506},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.038,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.411,"5min":1.4319999999999999,"15min":1.4319999999999999},"last":1.302}]},{"osd":6,"last update":"Sat Mar 13 16:51:37 2021","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":1.492},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"last":1.218}]},{"osd":7,"last update":"Sat Mar 13 16:52:41 2021","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":1.032,"15min":1.034},"min":{"1min":0.66800000000000004,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.395,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.069,"15min":1.075},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.355,"5min":1.591,"15min":1.591},"last":1.26}]}]},{"osd":3,"up_from":0,"seq":55834574886,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":0,"seq":55834574886,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":694896,"kb_used_data":660904,"kb_used_omap":0,"kb_used_meta":33984,"kb_avail":93676944,"statfs":{"total":96636764160,"available":95925190656,"internally_reserved":0,"allocated":676765696,"data_stored":686126081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34799616},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.36099999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.131,"5min":1.29,"15min":1.29},"last":1.202},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.67800000000000005,"15min":0.66800000000000004},"min":{"1min":0.41299999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":1.2110000000000001}]},{"osd":1,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.71199999999999997,"15min":0.70199999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":1.1419999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.77900000000000003,"15min":0.77200000000000002},"min":{"1min":0.497,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.2150000000000001,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":1.0920000000000001}]},{"osd":2,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.89800000000000002,"15min":0.89700000000000002},"min":{"1min":0.47699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.272,"5min":1.456,"15min":1.456},"last":1.294},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.89500000000000002,"15min":0.88300000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.6020000000000001},"last":1.272}]},{"osd":3,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.153,"5min":1.204,"15min":1.204},"last":1.333},{"interface":"front","average":{"1min":0.85299999999999998,"5min":0.80900000000000005,"15min":0.80100000000000005},"min":{"1min":0.54100000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.385,"5min":1.385,"15min":1.385},"last":1.0589999999999999}]},{"osd":4,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.78300000000000003,"15min":0.76900000000000002},"min":{"1min":0.53200000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.76700000000000002,"15min":0.76000000000000001},"min":{"1min":0.47599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.139,"5min":1.204,"15min":1.204},"last":0.85899999999999999}]},{"osd":6,"last update":"Sat Mar 13 16:51:37 2021","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.4550000000000001,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.84699999999999998}]},{"osd":7,"last update":"Sat Mar 13 16:52:37 2021","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.89300000000000002,"15min":0.89200000000000002},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.298,"5min":1.393,"15min":1.393},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.93700000000000006,"15min":0.93300000000000005},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2569999999999999,"5min":1.514,"15min":1.514},"last":0.76600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":820244480,"data_stored":829751296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":738680832,"data_stored":744456192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":678408192,"data_stored":686342144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105893888,"data_stored":113971200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":674791424,"data_stored":685076480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":598249472,"data_stored":604114944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1091485696,"data_stored":1107042304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} , 2021-03-13T16:53:27.706 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set rbd pgp_num 8 2021-03-14T06:23:33.018 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:51.884 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 , 2021-03-13T16:53:51.978+0000 7f7666da7700 -1 WARNING: all dangerous and experimental features are enabled. , 2021-03-13T16:53:51.983+0000 7f7666da7700 -1 WARNING: all dangerous and experimental features are enabled. 2021-03-14T06:23:33.021 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:53:58.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574892 got 55834574893 for osd.7 , 2021-03-13T16:53:58.635 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete , 2021-03-13T16:53:58.636 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-03-14T06:23:33.022 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:54:09.285 DEBUG:teuthology.orchestra.run.gibba040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 , 2021-03-13T16:54:09.372+0000 7f178b61f700 -1 WARNING: all dangerous and experimental features are enabled. , 2021-03-13T16:54:09.377+0000 7f178a3bd700 -1 WARNING: all dangerous and experimental features are enabled. 2021-03-14T06:23:33.024 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:54:11.518 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2021-03-13T16:53:26.043923+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=53, tm_sec=26, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=54, tm_sec=7, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T16:54:11.518 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2021-03-13T16:54:05.566545+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=54, tm_sec=5, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=54, tm_sec=7, tm_wday=5, tm_yday=72, tm_isdst=0) , 2021-03-13T16:54:11.519 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2021-03-13T16:52:35.014193+0000 time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=52, tm_sec=35, tm_wday=5, tm_yday=72, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=3, tm_mday=13, tm_hour=16, tm_min=54, tm_sec=7, tm_wday=5, tm_yday=72, tm_isdst=0) 2021-03-14T06:23:33.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:54:53.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@gibba040.front.sepia.ceph.com , 2021-03-13T16:54:53.477 DEBUG:teuthology.orchestra.run.gibba040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 , 2021-03-13T16:54:53.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@gibba042.front.sepia.ceph.com 2021-03-14T06:23:33.025 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T16:55:35.729 INFO:teuthology.orchestra.run.gibba042.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -680us[ -680us] +/- 87ms , 2021-03-13T16:55:35.729 INFO:teuthology.orchestra.run.gibba042.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 53 -666us[ -666us] +/- 58ms , 2021-03-13T16:55:35.730 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:33.026 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.332 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.334 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.334 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.335 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.336 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.337 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.338 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.339 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.340 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.341 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.342 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.343 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.344 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.344 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.345 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.346 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.346 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.347 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.348 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.349 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.349 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.350 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.351 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.351 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.352 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.353 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.353 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.354 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.355 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.355 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.356 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.357 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.358 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.358 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.359 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.359 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.360 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.361 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.361 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.362 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.364 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.366 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.366 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.367 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.368 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.368 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.370 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.371 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.427 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.429 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.429 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.430 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.431 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.432 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.433 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.434 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.435 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.436 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.437 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.438 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.439 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.440 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.441 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.442 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.443 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.444 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.445 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.446 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.447 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.448 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.449 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.450 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.451 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.452 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.453 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.454 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.455 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.456 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.457 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.459 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.461 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.461 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.462 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.463 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.463 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.464 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.465 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.517 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.518 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.519 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.520 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.520 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.521 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.522 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.522 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.523 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.524 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.524 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.525 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.525 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.526 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.527 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.527 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.528 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.528 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.529 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.530 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.530 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.531 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.532 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.532 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.533 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.533 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.534 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.535 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.535 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.536 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.536 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.537 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.538 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.538 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.539 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.540 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.540 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.541 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.542 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.543 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.544 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.545 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.547 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.548 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.549 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.550 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.550 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.551 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.552 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.552 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.608 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.609 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.610 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.611 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.611 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.612 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.613 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.613 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.614 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.615 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.615 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.616 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.616 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.617 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.618 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.619 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.619 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.620 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.621 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.621 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.622 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.622 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.623 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.624 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.624 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.625 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.625 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.626 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.627 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.627 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.628 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.629 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.629 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.630 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.630 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.631 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.632 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.632 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.633 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.634 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.635 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.636 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.636 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.637 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.638 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.640 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.640 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.641 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.642 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.643 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.643 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.644 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.699 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.700 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.701 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.701 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.702 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.703 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.704 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.704 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.705 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.705 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.706 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.707 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.707 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.708 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.709 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.709 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.710 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.710 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.711 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.712 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.712 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.713 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.713 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.714 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.715 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.715 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.716 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.716 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.717 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.718 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.718 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.719 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.720 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.720 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.721 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.721 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.722 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.723 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.723 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.724 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.725 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.725 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.726 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.727 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.727 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.729 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.730 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.731 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.732 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.732 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.733 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.734 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.734 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.794 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.795 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.796 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.797 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.797 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.798 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.799 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.799 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.800 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.801 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.801 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.802 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.803 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.803 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.804 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.805 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.806 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.807 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.807 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.808 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.809 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.810 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.811 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.812 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.813 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.814 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.815 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.815 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.816 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.817 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.818 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.819 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.820 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.821 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.822 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.823 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.824 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.826 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.827 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.828 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.828 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.829 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.830 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.831 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.831 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.886 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.887 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.888 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.888 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.889 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.890 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.891 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.892 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.893 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.894 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.895 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.896 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.897 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.897 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.898 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.899 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.899 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.900 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.900 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.901 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.902 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.902 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.903 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.904 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.904 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.905 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.906 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.906 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.907 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.908 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.908 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.909 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.910 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.910 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:34.911 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:34.912 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:34.912 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:34.913 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:34.913 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:34.914 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:34.915 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:34.916 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:34.918 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:34.919 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:34.919 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:34.920 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:34.921 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:34.921 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:34.922 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:34.975 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:34.976 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:34.977 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:34.978 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:34.978 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:34.979 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:34.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:34.980 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:34.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:34.981 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:34.982 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:34.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.983 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:34.984 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:34.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:34.985 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.986 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:34.986 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:34.987 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:34.988 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:34.988 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:34.989 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:34.989 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:34.990 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:34.991 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:34.991 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:34.992 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:34.993 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:34.993 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:34.994 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.995 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.995 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:34.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:34.996 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:34.997 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:34.998 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:34.999 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:34.999 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:35.000 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:35.000 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:35.001 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:35.002 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:35.002 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:35.003 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:35.004 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:35.005 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:35.007 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:35.007 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:35.008 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:35.009 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:35.009 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:35.010 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:35.011 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:35.069 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:35.070 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:35.071 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:35.072 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:35.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:35.073 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:35.074 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:35.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:35.075 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:35.076 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:35.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:35.077 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.078 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:35.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:35.079 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:35.080 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:35.081 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:35.082 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:35.082 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:35.083 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.084 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:35.084 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.085 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:35.086 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:35.086 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:35.087 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:35.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:35.088 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:35.089 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.090 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.090 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:35.091 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:35.092 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:35.093 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:35.094 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:35.094 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:35.095 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:35.096 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:35.097 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:35.097 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:35.098 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:35.099 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:35.099 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:35.101 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:35.102 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:35.103 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:35.104 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:35.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:35.105 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:35.106 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:35.107 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:35.554 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:35.555 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:35.556 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:35.556 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:35.557 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:35.558 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:35.558 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:35.559 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:35.560 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:35.561 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:35.561 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:35.562 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.563 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:35.563 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:35.564 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:35.565 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.565 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:35.566 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:35.567 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:35.567 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:35.568 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:35.569 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.570 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:35.571 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:35.571 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:35.572 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:35.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:35.573 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:35.574 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.575 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.575 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:35.576 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.577 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:35.577 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:35.578 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:35.579 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:35.580 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:35.580 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:35.581 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:35.582 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:35.582 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:35.583 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:35.584 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:35.585 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:35.586 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:35.588 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:35.589 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:35.590 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:35.590 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:35.591 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:35.592 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:35.592 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:35.654 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:35.655 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:35.656 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:35.657 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:35.657 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:35.658 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:35.659 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:35.660 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:35.660 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:35.661 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:35.661 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:35.662 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.663 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:35.663 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:35.664 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:35.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.665 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:35.666 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:35.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:35.667 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:35.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.668 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:35.669 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:35.670 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:35.671 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:35.672 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:35.673 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:35.673 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:35.674 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.675 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.676 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:35.676 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.677 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:35.678 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:35.678 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:35.679 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:35.680 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:35.681 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:35.681 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:35.682 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:35.682 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:35.683 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:35.684 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:35.684 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:35.686 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:35.689 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:35.690 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:35.691 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:35.692 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:35.693 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:35.693 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:35.694 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:35.749 WARNING:scrape:Ignoring malparsed backtrace: ceph version 17.0.0-1894-g95bec987 (95bec98739256c1e2cefa7b63b1eff66d400d46c) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fb077e90d2b] , 2: /usr/lib/ceph/libceph-common.so.2(+0x262f33) [0x7fb077e90f33] 2021-03-14T06:23:35.751 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:29.615 INFO:teuthology.orchestra.run.gibba039.stdout:7.4 , 2021-03-13T14:17:29.621 INFO:teuthology.orchestra.run.gibba039.stdout:3.1d , 2021-03-13T14:17:29.622 INFO:teuthology.orchestra.run.gibba039.stdout:7.9 2021-03-14T06:23:35.751 WARNING:scrape:Ignoring malparsed backtrace: File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 479, in wait , proc.wait() , File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 161, in wait 2021-03-14T06:23:35.752 WARNING:scrape:Ignoring malparsed backtrace: Read #4:f6792a7a:::obj_delete_at_hint.0000000097:head# , Read #4:f7e9c90e:::data_log.78:head# , Export successful 2021-03-14T06:23:35.753 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:38.693 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: cluster , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: 2021-03-13T14:17:37.654679+0000 mon.a , 2021-03-13T14:17:38.694 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:38 gibba039 conmon[26267]: (mon.0) 778 : cluster [DBG] osdmap e156: 8 total, 7 up, 8 in 2021-03-14T06:23:35.753 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:42.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:41 gibba039 conmon[26267]: audit 2021-03-13T14:17:41.561277+0000 mon.a (mon.0) 783 : audit [INF] from='client.? 172.21.2.137:0/2104646923' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch , 2021-03-13T14:17:42.428 DEBUG:teuthology.orchestra.run.gibba039:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph daemon osd.5 dump_historic_ops , 2021-03-13T14:17:42.822 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:17:42 gibba039 conmon[26267]: cluster 2021-03-13T14:17:41.535803+0000 osd.6 (osd.6) 27 : cluster [DBG] 4.1f scrub ok 2021-03-14T06:23:35.754 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:48.995 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:17:48 gibba039 podman[58743]: 2021-03-13 14:17:48.776699449 +0000 UTC m=+0.298317214 container attach 3e4df636dce3af9227862b74e0f1738bfbc4dd5712a464d43ccfd01df2cf6859 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-activate, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.256386+0000 mgr.y (mgr.14144) 385 : cluster [DBG] pgmap v384: 148 pgs: 26 active+undersized+degraded, 85 active+clean, 37 active+undersized; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.0 KiB/s rd, 1 op/s; 471/4188 objects degraded (11.246%) , 2021-03-13T14:17:48.997 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:48 gibba037 conmon[26229]: cluster 2021-03-13T14:17:48.536720+0000 osd.3 (osd.3) 51 : cluster [DBG] 4.0 scrub ok 2021-03-14T06:23:35.755 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: cephadm 2021-03-13T14 , 2021-03-13T14:17:53.250 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: :17:52.492037+0000 mgr.y (mgr.14144) 391 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable , 2021-03-13T14:17:53.251 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:17:52 gibba037 conmon[26229]: audit 2021-03-13 2021-03-14T06:23:35.756 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: audit 2021-03-13T14:17:55.821541+0000 mon.b (mon , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: .2) 27 : audit [INF] from='osd.4 [v2:172.21.2.139:6800/697312535,v1:172.21.2.139:6801/697312535]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0820, "args": ["host=gibba039", "root=default"]}]: dispatch , 2021-03-13T14:17:57.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:56 gibba037 conmon[32270]: cluster 2021-03-13T14:17:56. 2021-03-14T06:23:35.756 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: [DBG] purged_snaps scrub ok , 2021-03-13T14:17:58.257 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: audit 2021-03-13T14:17:57.108345 , 2021-03-13T14:17:58.258 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:17:57 gibba037 conmon[32270]: +0000 mon.a (mon.0) 797 : audit [INF] from='mgr.14144 172.21.2.137:0/972677597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-14T06:23:35.757 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 6.7 scrub ok , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: cluster 2021-03-13T14:18: , 2021-03-13T14:18:02.249 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:01 gibba037 conmon[26229]: 01.653501+0000 mon.a (mon.0) 801 2021-03-14T06:23:35.758 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:03.576 INFO:teuthology.orchestra.run.gibba039.stdout: "events": [ , 2021-03-13T14:18:03.577 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.758 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:07.077 INFO:teuthology.orchestra.run.gibba037.stdout:mon.a: {} , 2021-03-13T14:18:07.084 INFO:teuthology.orchestra.run.gibba037.stdout:mon.c: {} , 2021-03-13T14:18:07.099 INFO:teuthology.orchestra.run.gibba037.stdout:mon.b: {} 2021-03-14T06:23:35.759 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:11.110 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "ops": [], , 2021-03-13T14:18:11.111 INFO:teuthology.orchestra.run.gibba039.stdout: "complaint_time": 30, 2021-03-14T06:23:35.759 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:13.256 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:12 gibba037 conmon[32270]: +0000 mon.a (mon.0) 809 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout:{ , 2021-03-13T14:18:14.204 INFO:teuthology.orchestra.run.gibba039.stdout: "size": 20, 2021-03-14T06:23:35.760 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.886097+0000", , 2021-03-13T14:18:14.227 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0022741710000000002 , 2021-03-13T14:18:14.228 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.761 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.249 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002707882, , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "type_data": { , 2021-03-13T14:18:14.250 INFO:teuthology.orchestra.run.gibba039.stdout: "flag_point": "started", 2021-03-14T06:23:35.761 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.272 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "done", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.921668+0000", , 2021-03-13T14:18:14.273 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.002638367 2021-03-14T06:23:35.762 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.295 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "reached_pg", , 2021-03-13T14:18:14.296 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:57.950462+0000", 2021-03-14T06:23:35.763 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.318 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", 2021-03-14T06:23:35.763 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.00027261700000000001 , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: }, , 2021-03-13T14:18:14.341 INFO:teuthology.orchestra.run.gibba039.stdout: { 2021-03-14T06:23:35.764 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 0.0052807169999999999 , 2021-03-13T14:18:14.363 INFO:teuthology.orchestra.run.gibba039.stdout: } , 2021-03-13T14:18:14.364 INFO:teuthology.orchestra.run.gibba039.stdout: ] 2021-03-14T06:23:35.764 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:58.339413+0000", , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 2.0941000000000002e-05 , 2021-03-13T14:18:14.386 INFO:teuthology.orchestra.run.gibba039.stdout: }, 2021-03-14T06:23:35.765 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "dispatched", , 2021-03-13T14:18:14.408 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:17:59.984083+0000", , 2021-03-13T14:18:14.409 INFO:teuthology.orchestra.run.gibba039.stdout: "duration": 3.3763999999999997e-05 2021-03-14T06:23:35.766 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "initiated", , 2021-03-13T14:18:14.431 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.217588+0000", 2021-03-14T06:23:35.766 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: { , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "event": "commit_sent", , 2021-03-13T14:18:14.453 INFO:teuthology.orchestra.run.gibba039.stdout: "time": "2021-03-13T14:18:00.220740+0000", 2021-03-14T06:23:35.767 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster 2021-03-13 , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: T14:18:14.271277+0000 mgr.y (mgr.14144) 404 : , 2021-03-13T14:18:15.249 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:14 gibba037 conmon[32270]: cluster [DBG] pgmap v406: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 852 B/s rd, 0 op/s 2021-03-14T06:23:35.768 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:18.996 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[26229]: [INF] from='client.? 172.21.2.137:0/537203305' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster 2021-03-13T14:18:17.365963+0000 osd.7 (osd.7) 55 : , 2021-03-13T14:18:18.997 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:18 gibba037 conmon[32270]: cluster [DBG] 3.11 scrub ok 2021-03-14T06:23:35.768 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:22.986 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.297102+0000 osd.7 (osd.7) 57 : , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster [DBG] 5.10 scrub ok , 2021-03-13T14:18:22.987 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:22 gibba037 conmon[32270]: cluster 2021-03-13T14:18:21.648812+0000 osd.3 (osd.3) 65 : cluster [DBG] 2021-03-14T06:23:35.769 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:28.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: cluster 2021-03-13T14:18:27.451560+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1d deep-scrub ok , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: audit 2021-03-13T14:18:27.476317+0000 mon.c (mon.1) 29 : audit [INF] , 2021-03-13T14:18:28.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:27 gibba039 conmon[26267]: from='client.? 172.21.2.137:0/3930479037' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.770 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:34.052 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:33 gibba037 conmon[32270]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch , 2021-03-13T14:18:34.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: cluster 2021-03-13T14:18:32.288463+0000 osd.0 (osd.0) 59 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:18:34.073 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:33 gibba039 conmon[26267]: audit 2021-03-13T14:18:33.693291+0000 mon.a (mon.0) 831 : audit [INF] from='client.? 172.21.2.137:0/3324846631' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.770 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03-13T14:18:38.833266+0000 mon.a (mon.0 , 2021-03-13T14:18:40.250 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: ) 837 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in , 2021-03-13T14:18:40.251 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:39 gibba037 conmon[32270]: cluster 2021-03- 2021-03-14T06:23:35.771 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:44.247 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: cluster 2021-03-13T14:18:42.818917+0000 osd.3 (osd.3) 72 : cluster [DBG] 6.c scrub ok , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: audit 2021-03-13T14:18:43.778844+0000 mon.c (mon.1) 31 : audit , 2021-03-13T14:18:44.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:43 gibba037 conmon[32270]: [INF] from='client.? 172.21.2.137:0/438838255' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-14T06:23:35.772 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:48.323 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: cluster 2021-03- , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: 13T14:18:47.425779+0000 osd.5 ( , 2021-03-13T14:18:48.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:47 gibba039 conmon[26267]: osd.5) 61 : cluster [DBG] 5.15 scrub ok 2021-03-14T06:23:35.772 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: 2021-03-13T14:18:51.105188+0000 osd.1 (osd.1) 53 : cluster , 2021-03-13T14:18:52.324 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: [DBG] 3.4 scrub ok , 2021-03-13T14:18:52.325 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:51 gibba039 conmon[26267]: cluster 2021 2021-03-14T06:23:35.773 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:57.998 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021-03-13T14:18:56.657255+0000 osd.4 (osd , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: .4) 23 : cluster [DBG] purged_snaps scrub ok , 2021-03-13T14:18:57.999 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:18:57 gibba037 conmon[26229]: cluster 2021- 2021-03-14T06:23:35.774 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: 2021-03-13T14:18 , 2021-03-13T14:18:59.752 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:18:59 gibba037 conmon[32270]: :59.366657+0000 mgr.y (mgr.14144) 434 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:00.072 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:18:59 gibba039 conmon[26267]: cluster 2021-03-13T14:18:58.388457+0000 osd.0 (osd.0) 69 : cluster [DBG] 5.1e deep-scrub ok 2021-03-14T06:23:35.774 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:07.999 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: 07.264787+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.17 scrub ok , 2021-03-13T14:19:08.000 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:07 gibba037 conmon[32270]: audit 2021-03-13T14:19:07.599774+0000 mon.c (mon.1) 33 : 2021-03-14T06:23:35.775 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: cluster 2021-03-13T14:19 , 2021-03-13T14:19:14.326 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: :13.249430+0000 osd.5 (osd.5) 67 : cluster [DBG] , 2021-03-13T14:19:14.327 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:13 gibba039 conmon[26267]: 4.8 scrub ok 2021-03-14T06:23:35.776 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: :22.307998+0000 mgr.y (mgr.14144) 448 : cluster [DBG] , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[32270]: pgmap v467: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.1 KiB/s rd, 1 op/s , 2021-03-13T14:19:23.074 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:22 gibba037 conmon[26229]: cluster 2021-03-13T14:19:22.093320+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.8 deep-scrub ok 2021-03-14T06:23:35.777 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:30.322 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:29 gibba039 conmon[26267]: audit 2021-03-13T14:19:29.398860+0000 mgr.y (mgr.14144) 452 : audit [DBG] from='client.34295 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch , 2021-03-13T14:19:30.649 INFO:teuthology.orchestra.run.gibba037.stdout:231928234055 , 2021-03-13T14:19:30.650 DEBUG:teuthology.orchestra.run.gibba037:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:95bec98739256c1e2cefa7b63b1eff66d400d46c shell --fsid 9a120794-8405-11eb-908c-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-14T06:23:35.777 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:36.154 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.651036+0000 osd.6 (osd.6) 47 : cluster [DBG] 5.11 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:34.927446+0000 osd.3 (osd.3) 83 : cluster [DBG] 5.0 scrub ok , 2021-03-13T14:19:36.155 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:36 gibba037 conmon[32270]: cluster 2021-03-13T14:19:35.193951+0000 2021-03-14T06:23:35.778 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: cluster 2021-03-13T14:19: , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: 40.317639+0000 mgr.y (mgr.14144) 459 : cluster [DBG] , 2021-03-13T14:19:41.347 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:41 gibba037 conmon[32270]: pgmap v478: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-14T06:23:35.779 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:48.499 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: osd.5) 78 : cluster [DBG] 6.a scrub ok , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: cluster 2021- , 2021-03-13T14:19:48.500 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:48 gibba037 conmon[32270]: 03-13T14:19:47.795944+0000 osd.7 (osd.7) 74 2021-03-14T06:23:35.779 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:52.248 INFO:journalctl@ceph.mon.c.gibba037.stdout:Mar 13 14:19:52 gibba037 conmon[32270]: 3.1 scrub ok , 2021-03-13T14:19:52.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:52 gibba039 conmon[26267]: cluster 2021-03-13T14:19:50.366186+0000 osd.0 (osd.0) 85 : cluster [DBG] 3.1 scrub ok , 2021-03-13T14:19:53.459 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:53 gibba037 conmon[26229]: cluster 2021-03-13T14:19:52.325025+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v486: 148 pgs: 148 active+clean; 2.9 GiB data, 4.3 GiB used, 668 GiB / 672 GiB avail; 1023 B/s rd, 0 op/s 2021-03-14T06:23:35.781 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:56.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: T14:19:55.292852+0000 osd.4 (osd.4) 40 : cluster [DBG] 5.1c deep-scrub ok , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: cluster 2021-03-13T14 , 2021-03-13T14:19:56.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:19:56 gibba039 conmon[26267]: :19:55.681578+0000 osd.6 (osd.6) 50 : cluster 2021-03-14T06:23:35.783 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:19:59.331 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.356467+0000 osd.0 (osd.0) 87 : cluster [DBG] 4.3 scrub ok , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.879826+0000 osd.1 (osd.1) 68 : cluster [DBG] purged_snaps scrub starts , 2021-03-13T14:19:59.332 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:19:59 gibba037 conmon[26229]: cluster 2021-03-13T14:19:57.880004+0000 osd.1 (osd.1) 69 : cluster 2021-03-14T06:23:35.783 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: cluster 2021-03-13T14:20:01.750027+0000 , 2021-03-13T14:20:02.574 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:02 gibba039 conmon[26267]: osd.6 (osd.6) 51 : cluster [DBG] 7.d scrub ok , 2021-03-13T14:20:03.231 INFO:journalctl@ceph.mon.a.gibba037.stdout:Mar 13 14:20:03 gibba037 conmon[26229]: cluster 2021-03-13T14:20:02.330093+0000 mgr.y ( 2021-03-14T06:23:35.784 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:06.304 DEBUG:teuthology.orchestra.run.gibba039:> sudo systemctl stop ceph-9a120794-8405-11eb-908c-001a4aab830c@mon.b , 2021-03-13T14:20:06.572 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 systemd[1]: Stopping Ceph mon.b for 9a120794-8405-11eb-908c-001a4aab830c... , 2021-03-13T14:20:06.573 INFO:journalctl@ceph.mon.b.gibba039.stdout:Mar 13 14:20:06 gibba039 conmon[26267]: debug 2021-03-13T14:20:06.391+0000 7fc1caa90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-14T06:23:35.785 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:20.058 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.003894419 +0000 UTC m=+0.150622004 container create 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.120506473 +0000 UTC m=+0.267233974 container init 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, GIT_CLEAN=True, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, RELEASE=wip-stretch-fixes-312-95bec98, org.label-schema.name=CentOS Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) , 2021-03-13T14:20:20.322 INFO:journalctl@ceph.osd.4.gibba039.stdout:Mar 13 14:20:20 gibba039 podman[62571]: 2021-03-13 14:20:20.16211008 +0000 UTC m=+0.308837583 container start 46cdf3026a0887e801c17322fc15be700b0fdb0a7d3b80adc42386af2a6db697 (image=quay.ceph.io/ceph-ci/ceph@sha256:f2ee2ba8803d25e04b40173e235a5e7ecc8a6182b7dec6abc04d1918086c4412, name=ceph-9a120794-8405-11eb-908c-001a4aab830c-osd.4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, io.buildah.version=1.18.0, RELEASE=wip-stretch-fixes-312-95bec98, maintainer=Dimitri Savineau ) 2021-03-14T06:23:35.785 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:20:47.622 DEBUG:teuthology.orchestra.run.gibba037:> sudo tar cz -f - -C /var/log/ceph -- . , 2021-03-13T14:20:49.451 DEBUG:teuthology.misc:Transferring archived files from gibba039:/var/log/ceph to /home/teuthworker/archive/gregf-2021-03-13_08:24:54-rados-wip-stretch-fixes-312-distro-basic-gibba/5961564/remote/ubuntu@gibba039.front.sepia.ceph.com/log , 2021-03-13T14:20:49.453 DEBUG:teuthology.orchestra.run.gibba039:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-14T06:23:35.786 WARNING:scrape:Ignoring malparsed backtrace: 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv03.front.sepi 74.6.168.72 3 u 57 64 377 0.121 3.976 3.439 , 2021-03-13T14:21:09.250 INFO:teuthology.orchestra.run.gibba039.stdout:+hv04.front.sepi 96.245.170.99 3 u 50 64 377 0.092 0.872 1.198 , 2021-03-13T14:21:09.251 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-14T06:23:35.787 WARNING:scrape:Ignoring malparsed backtrace: During handling of the above exception, another exception occurred: , , Traceback (most recent call last): 2021-03-14T06:23:38.038 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/results.py", line 34, in main int(args['--timeout']), args['--dry-run']) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/results.py", line 69, in results Scraper(archive_dir).analyze() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/scrape.py", line 490, in analyze if reason.match(job): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/scrape.py", line 204, in match job.get_last_tlog_line()).ratio() File "/usr/lib/python3.6/difflib.py", line 644, in ratio matches = sum(triple[-1] for triple in self.get_matching_blocks()) File "/usr/lib/python3.6/difflib.py", line 467, in get_matching_blocks la, lb = len(self.a), len(self.b) TypeError: object of type 'NoneType' has no len()